NLTK Assignment– 7

1. Create a unigram class that wraps FreqDist and exposes prob(w) for MLE and Laplace-smoothed probabilities. Use it on Brown news and print prob('india') under both MLE and Laplace.

2. Build interpolated trigram probabilities on Brown news:
   P(w3∣w1,w2)=λ3⋅MLEtrigram+λ2⋅MLEbigram+λ1⋅MLEunigramP(w_3|w_1,w_2)=\lambda_3\cdot \text{MLE}_{\text{trigram}}+\lambda_2\cdot \text{MLE}_{\text{bigram}}+\lambda_1\cdot \text{MLE}_{\text{unigram}}P(w3​∣w1​,w2​)=λ3​⋅MLEtrigram​+λ2​⋅MLEbigram​+λ1​⋅MLEunigram​ with (λ3,λ2,λ1)=(0.6,0.3,0.1)(\lambda_3,\lambda_2,\lambda_1)=(0.6,0.3,0.1)(λ3​,λ2​,λ1​)=(0.6,0.3,0.1).
   Compute this value for contexts ("in","the") → 'city', ("of","the") → 'government', and ("to","the") → 'market' and print the three probabilities (round to 6 decimals).

3. Implement sentence generation from the interpolated model (2) with max length 30, seeded by ("in","the"); sample at each step with random.Random(42) and stop if a period '.' is generated. Print the generated sentence.

4. Using Brown news, compute a ConditionalFreqDist of trigrams mapping (w1,w2) to w3 counts. For contexts ("in","the") and ("at","the"), print the top 10 w3 continuations with counts.

5. On Brown news (lowercase alphabetic), compute trigram PMI using TrigramAssocMeasures.pmi via TrigramCollocationFinder. Filter tokens to length ≥ 3 and remove stopwords. Print the top 20 trigrams with PMI (round to 3 decimals).

6. For the toy corpus, compute bigram chi-square collocations and print the top 10 with scores. Also print whether ('to','delhi') appears in the ranked list.

7. Implement a Jelinek–Mercer interpolation for bigrams on Brown news:
   P(w2∣w1)=α⋅MLEbigram+(1−α)⋅MLEunigramP(w_2|w_1)=\alpha\cdot \text{MLE}_{\text{bigram}}+(1-\alpha)\cdot \text{MLE}_{\text{unigram}}P(w2​∣w1​)=α⋅MLEbigram​+(1−α)⋅MLEunigram​ with $\alpha =0.7$.
   Compute this for contexts 'in'→'india', 'to'→'the', and 'of'→'government'.

8. Measure perplexity of two sentences under the Brown bigram Laplace-smoothed model:
   a) "aarav works at tcs in mumbai"
   b) "priya studies data science in pune"
   Print both values (scientific notation) and state which sentence is less perplexing.

9. Using the toy corpus, build a simple Markov text generator that alternates between bigram sampling and a forced newline token '<br>' after every 8 generated words. Generate 24 words (3 line blocks) seeded with 'aarav' and print the formatted output with line breaks.

10. On Brown news (lowercase alphabetic), compute the top 30 bigram collocations by PMI and then recompute after stopword removal. Print the overlap count and list 10 bigrams that dropped out when stopwords were removed.

11. Create a backoff bigram model for Brown: if MLE_bigram(context,next)==0, fall back to Lidstone unigram with γ=0.2. Compute P('india'|'in'), P('market'|'to'), and P('government'|'of').

12. Train a toy intent classifier for a chatbot using unigram presence over the following labeled utterances (train on all of them):
    ('greet', ['hi','hello','namaste','hello aarav','hi priya']),
('bye', ['bye','see you','good night','thanks bye']),
('thanks', ['thanks','thank you so much','thanks aarav']),
('refund', ['how to get train ticket refund','refund for cancelled irctc ticket','aarav needs ticket refund information']),
('bank', ['how to open bank account in delhi','documents for savings account','minimum balance for account']),
('weather', ['weather in mumbai today','is it raining in pune','temperature in bengaluru'])
    Build a FreqDist-based vocabulary, a boolean feature extractor, classify with nltk.NaiveBayesClassifier, and print predicted intent for these 4 inputs: "hello priya", "ticket refund rules", "open bank account delhi", "rain in pune".

13. Add a bigram language-model fallback to the chatbot from 12: if classifier confidence < 0.6, generate a 15-word bigram sample from the toy corpus seeded by 'priya' and reply with I can share more: <generated text>. Demonstrate on the input "tell me something".

14. Implement a pattern-to-template response table for chatbot intents (e.g., greet → “Namaste! How can I help?”; refund → “IRCTC refunds depend on timing; do you want steps?”; bank → “To open a bank account in Delhi, you need KYC documents.”; weather → “Please specify city.”; thanks → “Happy to help!”; bye → “Goodbye!”). Using predictions from 12, print the chosen response for each input.

15. Build a tiny retrieval module: given the 6 toy documents from Assignment 6, Basic Q20 (repeat them in code), compute document unigram probabilities (MLE) and a query likelihood with Jelinek–Mercer $\alpha =0.7$ against each document model using the query "aarav needs train ticket refund information" (lowercase). Print the top 3 documents.

16. For Brown news, compute pointwise mutual information (PMI) for the specific bigram ("new","delhi") and print its PMI value (if the bigram does not occur, print "not found").

17. On Brown news, compute the t-score for the bigram ("of","india") and print the statistic (or "not found" if absent). Clearly show counts used: f(w1,w2), f(w1), f(w2), and N.

18. Using the toy corpus, train separate unigram models per intent class from 12 by pooling utterances for each label. Classify the query "documents needed for account in delhi" by computing the log-likelihood under each class model + log prior (uniform prior) and print the chosen class.

19. Combine collocation features with the intent classifier: add binary features for the presence of the top 5 bigram collocations (by PMI) mined from all training utterances in 12; retrain NB and print the new predictions for the same 4 inputs from 12.

20. Save to disk (pickle) the following chatbot artifacts: vocabulary, NB classifier, bigram generator ConditionalProbDist built on the toy corpus, and the response template dict. Load them back and run a live test on "open savings account mumbai"; print predicted intent and response (and if low-confidence, the generated fallback text).

Advanced Questions

1. Build K-fold (k=5) splits of Brown news tokens to evaluate bigram models by perplexity on held-out folds (Laplace smoothing). Print the mean perplexity across folds for the sentence "india plans new policy in the market" (lowercase).

2. Implement interpolated trigram generation with (λ3,λ2,λ1)=(0.6,0.3,0.1)(\lambda_3,\lambda_2,\lambda_1)=(0.6,0.3,0.1)(λ3​,λ2​,λ1​)=(0.6,0.3,0.1) on Brown news and add a repeat-penalty: if the sampled token equals the previous token, resample once. Generate 50 tokens starting from ("in","the") with random.Random(42) and print the result.

3. Compare Laplace vs Lidstone (γ=0.2) smoothing for Brown bigrams by computing the log-probability of the sentence "priya studies data science in pune" under both; print both log-probs and the absolute difference.

4. Build a trigram backoff model: try trigram MLE; if unseen, back off to bigram Lidstone γ=0.2; if still unseen, back off to unigram Lidstone γ=0.2. Compute the probability of the sequence "aarav works at tcs in mumbai" and print the value.

5. Using Brown news, compute top 20 trigrams by PMI and then extract those containing an Indian city token from the set {mumbai, delhi, pune, bengaluru, chennai, hyderabad} (lowercase). Print all matches (or "none" if empty).

6. Extend the chatbot to intent + slot filling with simple regexes:
   a) for refund, extract {"mode": "train|flight", "city": <CityName>} if mentioned;
   b) for bank, extract {"city": <CityName>};
   c) for weather, extract {"city": <CityName>}.
   Test on: "ticket refund in mumbai", "open bank account delhi", "weather in pune today"; print predicted intent and extracted slots.

7. Create a response ranking step: for inputs that map to refund, rank three candidate responses by query likelihood under a bigram model trained on all refund-labeled utterances (from your earlier dataset). Candidates:
   a) "IRCTC refunds depend on timing and class."
   b) "You can apply for ticket refund online."
   c) "Refunds are not allowed for all tickets."
   Print the winning response for "aarav needs ticket refund information".

8. Perform a collocation significance comparison on Brown news by computing the rank positions of a target bigram (choose ("railway","ticket"), or print "not found" if absent) under PMI, t-test, and chi-square; print the three ranks side by side (or a clear message if missing).

9. Build a document-aware fallback: if the chatbot confidence < 0.6 and the user mentions any of {'refund','ticket','account','bank','weather'}, instead of text generation, return the best matching document from the 6 toy docs using the query likelihood method (as defined earlier). Demonstrate on "need ticket refund rules" and "account opening delhi" and print the returned titles (first 6 words of each document).

10. Package a mini LM toolkit lm_utils.py exposing:
    a) build_unigram(tokens) -> FreqDist
    b) build_cfd_bigrams(tokens) -> ConditionalFreqDist
    c) cpd_from_cfd(cfd, smoother='mle'|'laplace'|'lidstone', gamma=0.2)
    d) generate_bigram(cpd, seed, n, rng)
    e) p_sentence_bi(tokens, cpd, unk_unigram=None) (optional backoff)
    f) top_pmi_bigrams(tokens, stop=None, n=20)
    Demonstrate by training on Brown news, generating 20 tokens from seed 'the', printing PMI top 10, and computing bigram probability of "india plans policy" (lowercase, alphabetic).