3기_4주차_김윤진

[yxzkng]

아마존 리뷰를 가지고 요약, 번역, 카테고리 분류를 진행하였습니다.

Summary by CodeRabbit

• New Features

  • Added end-to-end NLP workflows: summarization, translation, and zero-shot classification for Amazon reviews.
  • Introduced a KLUE-MRC pipeline with context summarization, English translation, and emotion analysis.
  • Included dataset previews and sample outputs to inspect intermediate and final results.
  • Enabled GPU-aware execution with updated accelerator metadata for faster runs.

• Documentation

  • Expanded notebooks with executable examples and step-by-step cells demonstrating data loading, transformation, and model inference.

[coderabbitai]

Warning

Rate limit exceeded

@yunjin-Kim4809 has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 2 minutes and 12 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 1e2eb43 and 4f514bb.

📒 Files selected for processing (2)

• assignment/huggingface.ipynb (1 hunks)
• assignment/huggingface.txt (1 hunks)

Walkthrough

Adds a new assignment/huggingface.py with a linear dataset → summarization → translation → classification workflow. Updates huggingface_basics.ipynb primarily in metadata and minor cell changes. Expands transformer.ipynb to implement a full KLUE-MRC pipeline with summarization, translation, and emotion analysis, including device handling and installation cells.

Changes

Cohort / File(s)	Summary of edits
New HF pipeline script assignment/huggingface.py	New notebook-like JSON script installing deps, loading amazon_polarity, summarizing with BART, translating to Korean via NLLB-200, and zero-shot classifying with BART-MNLI; adds columns and prints previews.
Notebook metadata and minor cell tweaks huggingface_basics.ipynb	Adjusted metadata, execution info, and light in-cell updates; no public interfaces changed.
KLUE-MRC end-to-end pipeline transformer.ipynb	Replaces TODOs with full workflow: load KLUE-MRC subset, summarize (KoBART), translate (NLLB-200), emotion analysis (go_emotions), map-based transforms, device handling, install cells, and result previews.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  participant User
  participant Script as assignment/huggingface.py
  participant Datasets as HF Datasets
  participant Summ as Summarization Pipeline (BART)
  participant Trans as Translation Pipeline (NLLB-200)
  participant ZS as Zero-shot Classifier (BART-MNLI)

  User->>Script: Run
  Script->>Datasets: load(amazon_polarity)
  Datasets-->>Script: dataset subset
  Script->>Summ: summarize(review_text)
  Summ-->>Script: summary
  Script->>Trans: translate(summary -> Korean)
  Trans-->>Script: ko_summary
  Script->>ZS: classify(summary, labels)
  ZS-->>Script: predicted label
  Script-->>User: dataset with summary, ko_summary, label

Loading

sequenceDiagram
  autonumber
  participant User
  participant NB as transformer.ipynb
  participant Datasets as HF Datasets (KLUE-MRC)
  participant S as Summarizer (KoBART)
  participant T as Translator (NLLB-200)
  participant E as Emotion (go_emotions)

  User->>NB: Execute cells
  NB->>Datasets: load(klue_mrc subset)
  Datasets-->>NB: dataset
  loop per example
    NB->>S: summarize(context)
    S-->>NB: summary
    NB->>T: translate(summary: ko -> en)
    T-->>NB: english_summary
    NB->>E: analyze(english_summary or summary)
    E-->>NB: emotion label
  end
  NB-->>User: final_dataset with summary, english_summary, emotion

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

I thump my paws on silicon ground,
Pip wheels spin, new models found! 🐇
Summaries bloom, then hop to Hangul,
Emotions peek—a carrot-angle. 🥕
From KLUE trails to stars above,
I ship these cells with rabbit love.

Pre-merge checks and finishing touches

❌ Failed checks (1 inconclusive)

Check name	Status	Explanation	Resolution
Title Check	❓ Inconclusive	The title “3기_4주차_김윤진” is a generic identifier for the assignment and author rather than a concise summary of the substantive changes, so it does not convey the main purpose or content of the pull request. It fails to inform reviewers about the addition of summarization, translation, and classification pipelines using Hugging Face tools on Amazon reviews. As a result, it is too vague to serve as an effective summary of the work.	Please rename the pull request to a concise, descriptive title such as “Add Amazon review summarization, translation, and classification pipeline” to clearly convey the primary changes.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changes. Docstring coverage check skipped.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (3)

huggingface_basics.ipynb (2)

327-332: Replace bare expressions with explicit print/display.

Cells use values as standalone expressions (e.g., config_dict, tokenizer([...]), klue_mrc_dataset) that don’t render reliably and trigger B018. Prefer print(...) or display(...) so the intent is explicit.

Example fixes:

• Lines 327-332:

- config_dict = base_model_config.__dict__
- config_dict
+ config_dict = base_model_config.__dict__
+ print(config_dict)

• Lines 681-686:

- tokenizer(['첫 번째 문장', '두 번째 문장']) # 두 문장은 각각 토큰화 된다.
+ print(tokenizer(['첫 번째 문장', '두 번째 문장'])) # 두 문장은 각각 토큰화 된다.

• Lines 1095-1096, 1120-1121:

- klue_mrc_dataset
+ print(klue_mrc_dataset)
- klue_mrc_dataset_only_train
+ print(klue_mrc_dataset_only_train)

As per static analysis hints.

Also applies to: 417-422, 507-511, 681-686, 719-724, 956-958, 1095-1096, 1120-1121

---

1220-1233: Remove duplicate imports to avoid F811.

pandas as pd is re-imported. Keep a single import at the top of the notebook.

-from datasets import load_dataset
-import pandas as pd
+# (imports above already present; remove duplicates here)

As per static analysis hints.

transformer.ipynb (1)

38-47: Import display explicitly to satisfy linters.

Add IPython display import so display(...) is defined outside Colab.

 from transformers import pipeline
 import torch
 import pandas as pd # 데이터 확인용
+from IPython.display import display

As per static analysis hints.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 1fcf7c2 and 1e2eb43.

📒 Files selected for processing (3)

• assignment/huggingface.py (1 hunks)
• huggingface_basics.ipynb (33 hunks)
• transformer.ipynb (11 hunks)

🧰 Additional context used

🪛 Ruff (0.13.3)

transformer.ipynb

107-107: Undefined name display

(F821)

---

174-174: Undefined name display

(F821)

huggingface_basics.ipynb

19-19: Found useless expression. Either assign it to a variable or remove it.

(B018)

---

123-123: Found useless expression. Either assign it to a variable or remove it.

(B018)

---

126-126: Found useless expression. Either assign it to a variable or remove it.

(B018)

---

163-163: Redefinition of unused pd from line 137

Remove definition: pd

(F811)

assignment/huggingface.py

1-1: Found useless expression. Either assign it to a variable or remove it.

(B018)

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

This .py file is a dumped notebook JSON; not executable and effectively a no-op.

• It’s a single top-level dict literal (B018 “useless expression”); nothing runs.
• Contains Colab !pip and display(...) in strings; not executed in Python.
• Bundles verbose pip logs and a KeyboardInterrupt. This doesn’t belong in VCS code.

Please either:

• Commit it as a real notebook (assignment/huggingface.ipynb) or
• Replace with a runnable Python script that implements the summarization → translation → classification flow.

Suggested minimal script replacement:

-{"nbformat":4,"nbformat_minor":0,"metadata":{...}}"cells":[...]}
+import re
+import torch
+import pandas as pd
+from datasets import load_dataset
+from transformers import pipeline
+
+device = 0 if torch.cuda.is_available() else -1
+
+# 1) Load small subset
+ds = load_dataset("amazon_polarity", split="train[:30]").remove_columns(["label"])
+
+# 2) Pipelines
+summarizer = pipeline("summarization", model="facebook/bart-large-cnn", device=device)
+translator = pipeline("translation", model="facebook/nllb-200-distilled-600M", device=device)
+classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")
+
+def clean_text(text: str) -> str:
+    text = re.sub(r'[\\^\\_]{2,}', ' ', text)
+    text = re.sub(r'([a-z])\\1{2,}', r'\\1\\1', text, flags=re.IGNORECASE)
+    return re.sub(r'\\s+', ' ', text).strip()
+
+def do_summarize(example):
+    cleaned = clean_text(example["content"])
+    out = summarizer(cleaned, max_length=150, min_length=30, do_sample=False)
+    example["summary"] = out[0]["summary_text"]
+    return example
+
+def do_translate(example):
+    out = translator(
+        example["summary"],
+        src_lang="eng_Latn",
+        tgt_lang="kor_Hang",
+        max_length=150,
+        min_length=30,
+        do_sample=False,
+    )
+    example["translation"] = out[0]["translation_text"]
+    return example
+
+CANDIDATE_LABELS = [
+    "Beauty","Fashion","Food","Beverages","Home Appliances","Electronics",
+    "Books","Health","Toys","Sports","Kitchenware","Music","Gaming",
+]
+
+def do_classify(example):
+    out = classifier(example["summary"], candidate_labels=CANDIDATE_LABELS)
+    example["category"] = out["labels"][0]
+    return example
+
+processed = ds.map(do_summarize).map(do_translate).map(do_classify)
+print(processed)
+print(pd.DataFrame(processed).head(10))

Also avoid !pip in code; pin dependencies in requirements/environment files instead.

As per static analysis hints.

Committable suggestion skipped: line range outside the PR's diff.

🧰 Tools

🪛 Ruff (0.13.3)

1-1: Found useless expression. Either assign it to a variable or remove it.

(B018)

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Fix emotion test output indexing.

For a single string with top_k=1, pipeline returns [{'label': ..., 'score': ...}] (not nested).

-test_emotion = emotion_classifier("I am very happy today!")
-print(f"감정 분석 테스트: {test_emotion[0][0]}")
+test_emotion = emotion_classifier("I am very happy today!")
+print(f"감정 분석 테스트: {test_emotion[0]}")

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	"test_emotion = emotion_classifier(\"I am very happy today!\")\n",
	"print(f\"감정 분석 테스트: {test_emotion[0][0]}\")"
	]
	test_emotion = emotion_classifier("I am very happy today!")
	print(f"감정 분석 테스트: {test_emotion[0]}")

🤖 Prompt for AI Agents

In transformer.ipynb around lines 330–332, the test output uses incorrect nested
indexing (test_emotion[0][0]) even though emotion_classifier with top_k=1
returns a flat list of dicts like [{'label':..., 'score':...}]; update the print
to access the first result's label (or the dict itself) e.g., use
test_emotion[0]['label'] when printing the predicted label (or test_emotion[0]
to print the full result) so indexing matches the actual return shape.

[coderabbitai]

⚠️ Potential issue | 🔴 Critical

Fix analyze_emotion result indexing (avoids KeyError/TypeError).

For single-text input with top_k=1 the result is a list[dict]. Extract the first dict.

-def analyze_emotion(example):
-  """데이터셋의 'english_summary'를 받아 감정 분석 결과를 반환하는 함수"""
-  emotion_result = emotion_classifier(example['english_summary'])
-    # 결과는 [[{'label': '...', 'score': ...}]] 형태 (top_k=1이므로 리스트 길이 1)
-  example['emotion'] = emotion_result[0][0]['label']
-  return example
+def analyze_emotion(example):
+  """데이터셋의 'english_summary'를 받아 감정 분석 결과를 반환하는 함수"""
+  result = emotion_classifier(example['english_summary'])[0]  # [{'label','score'}]
+  example['emotion'] = result['label']
+  return example

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	"def analyze_emotion(example):\n",
	" \"\"\"데이터셋의 'english_summary'를 받아 감정 분석 결과를 반환하는 함수\"\"\"\n",
	" # TODO: emotion_classifier 파이프라인을 사용하여 example['english_summary']의 감정을 분석하세요.\n",
	" # emotion_result = emotion_classifier(...)\n",
	" # top_k=1 이므로 결과는 [[{'label': '...', 'score': ...}]] 형태입니다.\n",
	" # TODO: 결과에서 가장 확률 높은 감정의 'label' 값만 추출하여 example 딕셔너리의 'emotion' 키 값으로 저장하세요.\n",
	" # example['emotion'] = ...\n",
	" emotion_result = emotion_classifier(example['english_summary'])\n",
	" # 결과는 [[{'label': '...', 'score': ...}]] 형태 (top_k=1이므로 리스트 길이 1)\n",
	" example['emotion'] = emotion_result[0][0]['label']\n",
	" return example\n",
	def analyze_emotion(example):
	"""데이터셋의 'english_summary'를 받아 감정 분석 결과를 반환하는 함수"""
	result = emotion_classifier(example['english_summary'])[0] # [{'label','score'}]
	example['emotion'] = result['label']
	return example

🤖 Prompt for AI Agents

In transformer.ipynb around lines 349 to 354, the analyze_emotion function
assumes emotion_classifier returns a nested list (emotion_result[0][0]['label'])
but for single-text input with top_k=1 the classifier returns a list of dicts
(list[dict]) causing KeyError/TypeError; change the extraction to handle the
actual shape by pulling the first dict (e.g., emotion_result[0]['label']), or
defensively check the structure and use the first element if it's a dict or the
first element of the first list if nested, then assign that label to
example['emotion'] and return the example.