How do you find duplicate words in a paragraph in python?

I can see where you are going with sort, as you can reliably know when you have hit a new word and keep track of counts for each unique word. However, what you really want to do is use a hash [dictionary] to keep track of the counts as dictionary keys are unique. For example:

words = sentence.split[]
counts = {}
for word in words:
    if word not in counts:
        counts[word] = 0
    counts[word] += 1

Now that will give you a dictionary where the key is the word and the value is the number of times it appears. There are things you can do like using collections.defaultdict[int] so you can just add the value:

counts = collections.defaultdict[int]
for word in words:
    counts[word] += 1

But there is even something better than that... collections.Counter which will take your list of words and turn it into a dictionary [an extension of dictionary actually] containing the counts.

counts = collections.Counter[words]

From there you want the list of words in sorted order with their counts so you can print them. items[] will give you a list of tuples, and sorted will sort [by default] by the first item of each tuple [the word in this case]... which is exactly what you want.

import collections
sentence = """As far as the laws of mathematics refer to reality they are not certain as far as they are certain they do not refer to reality"""
words = sentence.split[]
word_counts = collections.Counter[words]
for word, count in sorted[word_counts.items[]]:
    print['"%s" is repeated %d time%s.' % [word, count, "s" if count > 1 else ""]]

OUTPUT

"As" is repeated 1 time.
"are" is repeated 2 times.
"as" is repeated 3 times.
"certain" is repeated 2 times.
"do" is repeated 1 time.
"far" is repeated 2 times.
"laws" is repeated 1 time.
"mathematics" is repeated 1 time.
"not" is repeated 2 times.
"of" is repeated 1 time.
"reality" is repeated 2 times.
"refer" is repeated 2 times.
"the" is repeated 1 time.
"they" is repeated 3 times.
"to" is repeated 2 times.

Input : "Ravi had been saying that he had been there"
Output : had
 
Input : "Ravi had been saying that"
Output : No Repetition

Input : "he had had he"
Output : he

had

Many times, we have a need of analysing the text only for the unique words present in the file. So, we need to eliminate the duplicate words from the text. This is achieved by using the word tokenization and set functions available in nltk.

Without preserving the order

In the below example we first tokenize the sentence into words. Then we apply set[] function which creates an unordered collection of unique elements. The result has unique words which are not ordered.

import nltk
word_data = "The Sky is blue also the ocean is blue also Rainbow has a blue colour." 

# First Word tokenization
nltk_tokens = nltk.word_tokenize[word_data]

# Applying Set
no_order = list[set[nltk_tokens]]

print no_order

When we run the above program, we get the following output −

['blue', 'Rainbow', 'is', 'Sky', 'colour', 'ocean', 'also', 'a', '.', 'The', 'has', 'the']

Preserving the Order

To get the words after removing the duplicates but still preserving the order of the words in the sentence, we read the words and add it to list by appending it.

import nltk
word_data = "The Sky is blue also the ocean is blue also Rainbow has a blue colour." 
# First Word tokenization
nltk_tokens = nltk.word_tokenize[word_data]

ordered_tokens = set[]
result = []
for word in nltk_tokens:
    if word not in ordered_tokens:
        ordered_tokens.add[word]
        result.append[word]
     
print result        

When we run the above program, we get the following output −

['The', 'Sky', 'is', 'blue', 'also', 'the', 'ocean', 'Rainbow', 'has', 'a', 'colour', '.']

How do I find a repeating word in a string in python?

How do I remove repetitive words in Python?

How do you print repeated letters in a string in python?

How do you find duplicate words in a paragraph in Java?