This weblog publish particulars how I utilised GPT to translate the non-public memoir of a household pal, making it accessible to a broader viewers. Particularly, I employed GPT-3.5 for translation and Unstructured’s APIs for environment friendly content material extraction and formatting.
The memoir, a heartfelt account by my household pal Carmen Rosa, chronicles her upbringing in Bolivia and her romantic journey in Paris with an Iranian man through the vibrant Nineteen Seventies. Initially written in Spanish, we aimed to protect the essence of her narrative whereas increasing its attain to English-speaking readers by means of the appliance of LLM applied sciences.
Beneath you possibly can learn the interpretation course of in additional element or you possibly can access here the Colab Notebook.
I adopted the following steps for the interpretation of the guide:
- Import E-book Knowledge: I imported the guide from a Docx doc utilizing the Unstructured API and divided it into chapters and paragraphs.
- Translation Approach: I translated every chapter utilizing GPT-3.5. For every paragraph, I supplied the most recent three translated sentences (if accessible) from the identical chapter. This method served two functions:
- Fashion Consistency: Sustaining a constant type all through the interpretation by offering context from earlier translations.
- Token Restrict: Limiting the variety of tokens processed directly to keep away from exceeding the mannequin’s context restrict.
3. Exporting translation as Docx: I used Unstructured’s API as soon as once more to save lots of the translated content material in Docx format.
1. Libraries
We’ ll begin with the set up and import of the required libraries.
pip set up --upgrade openai
pip set up python-dotenv
pip set up unstructured
pip set up python-docx
import openai# Unstructured
from unstructured.partition.docx import partition_docx
from unstructured.cleaners.core import group_broken_paragraphs
# Knowledge and different libraries
import pandas as pd
import re
from typing import Record, Dict
import os
from dotenv import load_dotenv
2. Connecting to OpenAI’s API
The code beneath units up the OpenAI API key to be used in a Python undertaking. It’s essential to save your API key in an .env file.
import openai# Specify the trail to the .env file
dotenv_path = '/content material/.env'
_ = load_dotenv(dotenv_path) # learn native .env file
openai.api_key = os.environ['OPENAI_API_KEY']
3. Loading the guide
The code permits us to import the guide in Docx format and divide it into particular person paragraphs.
parts = partition_docx(
filename="/content material/libro.docx",
paragraph_grouper=group_broken_paragraphs
)
The code beneath returns the paragraph within the tenth index of parts.
print(parts[10])# Returns: Destino sorprendente, es el título que la autora le puso ...
4. Group guide into titles and chapters
The subsequent step includes creating an inventory of chapters. Every chapter can be represented as a dictionary containing a title and an inventory of paragraphs. This construction simplifies the method of translating every chapter and paragraph individually. Right here’s an instance of this format:
[
{"title": title 1, "content": [paragraph 1, paragraph 2, ..., paragraph n]},
{"title": title 2, "content material": [paragraph 1, paragraph 2, ..., paragraph n]},
...
{"title": title n, "content material": [paragraph 1, paragraph 2, ..., paragraph n]},
]
To attain this, we’ll create a operate known as group_by_chapter. Listed below are the important thing steps concerned:
- Extract Related Info: We will get every narrative textual content and title by calling
aspect.class. These are the one classes we’re desirous about translating at this level. - Determine Narrative Titles: We recognise that some titles needs to be a part of the narrative textual content. To account for this, we assume that italicised titles belong to the narrative paragraph.
def group_by_chapter(parts: Record) -> Record[Dict]:
chapters = []
current_title = Nonefor aspect in parts:
text_style = aspect.metadata.emphasized_text_tags # checks whether it is 'b' or 'i' and returns record
unique_text_style = record(set(text_style)) if text_style just isn't None else None
# we take into account a component a title if it's a title class and the type is daring
is_title = (aspect.class == "Title") & (unique_text_style == ['b'])
# we take into account a component a story content material if it's a narrative textual content class or
# if it's a title class, however it's italic or italic and daring
is_narrative = (aspect.class == "NarrativeText") | (
((aspect.class == "Title") & (unique_text_style is None)) |
((aspect.class == "Title") & (unique_text_style == ['i'])) |
((aspect.class == "Title") & (unique_text_style == ['b', 'i']))
)
# for brand spanking new titles
if is_title:
print(f"Including title {aspect.textual content}")
# Add earlier chapter when a brand new one is available in, except present title is None
if current_title just isn't None:
chapters.append(current_chapter)
current_title = aspect.textual content
current_chapter = {"title": current_title, "content material": []}
elif is_narrative:
print(f"Including Narrative {aspect.textual content}")
current_chapter["content"].append(aspect.textual content)
else:
print(f'### No have to convert. Aspect sort: {aspect.class}')
return chapters
Within the instance beneath, we are able to see an instance:
book_chapters[2] # Returns
{'title': 'Proemio',
'content material': [
'La autobiografía es considerada ...',
'Dentro de las artes literarias, ...',
'Se encuentra más próxima a los, ...',
]
}
5. E-book translation
To translate the guide, we observe these steps:
- Translate Chapter Titles: We translate the title of every chapter.
- Translate Paragraphs: We translate every paragraph, offering the mannequin with the most recent three translated sentences as context.
- Save Translations: We save each the translated titles and content material.
The operate beneath automates this course of.
def translate_book(book_chapters: Record[Dict]) -> Dict:
translated_book = []
for chapter in book_chapters:
print(f"Translating following chapter: {chapter['title']}.")
translated_title = translate_title(chapter['title'])
translated_chapter_content = translate_chapter(chapter['content'])
translated_book.append({
"title": translated_title,
"content material": translated_chapter_content
})
return translated_book
For the title, we ask GPT a easy translation as follows:
def translate_title(title: str) -> str:
response = shopper.chat.completions.create(
mannequin="gpt-3.5-turbo",
messages= [{
"role": "system",
"content": f"Translate the following book title into English:n{title}"
}]
)
return response.decisions[0].message.content material
To translate a single chapter, we offer the mannequin with the corresponding paragraphs. We instruct the mannequin as follows:
- Determine the position: We inform the mannequin that it’s a useful translator for a guide.
- Present context: We share the most recent three translated sentences from the chapter.
- Request translation: We ask the mannequin to translate the following paragraph.
Throughout this course of, the operate combines all translated paragraphs right into a single string.
# Perform to translate a chapter utilizing OpenAI API
def translate_chapter(chapter_paragraphs: Record[str]) -> str:
translated_content = ""for i, paragraph in enumerate(chapter_paragraphs):
print(f"Translating paragraph {i + 1} out of {len(chapter_paragraphs)}")
# Builds the message dynamically primarily based on whether or not there may be earlier translated content material
messages = [{
"role": "system",
"content": "You are a helpful translator for a book."
}]
if translated_content:
latest_content = get_last_three_sentences(translated_content)
messages.append(
{
"position": "system",
"content material": f"That is the most recent textual content from the guide that you have translated from Spanish into English:n{latest_content}"
}
)
# Provides the consumer message for the present paragraph
messages.append(
{
"position": "consumer",
"content material": f"Translate the next textual content from the guide into English:n{paragraph}"
}
)
# Calls the API
response = shopper.chat.completions.create(
mannequin="gpt-3.5-turbo",
messages=messages
)
# Extracts the translated content material and appends it
paragraph_translation = response.decisions[0].message.content material
translated_content += paragraph_translation + 'nn'
return translated_content
Lastly, beneath we are able to see the supporting operate to get the most recent three sentences.
def get_last_three_sentences(paragraph: str) -> str:
# Use regex to separate the textual content into sentences
sentences = re.break up(r'(?<!w.w.)(?<![A-Z][a-z].)(?<=.|?)s', paragraph)# Get the final three sentences (or fewer if the paragraph has lower than 3 sentences)
last_three = sentences[-3:]
# Be a part of the sentences right into a single string
return ' '.be part of(last_three)
6. E-book export
Lastly, we move the dictionary of chapters to a operate that provides every title as a heading and every content material as a paragraph. After every paragraph, a web page break is added to separate the chapters. The ensuing doc is then saved domestically as a Docx file.
from docx import Docdef create_docx_from_chapters(chapters: Dict, output_filename: str) -> None:
doc = Doc()
for chapter in chapters:
# Add chapter title as Heading 1
doc.add_heading(chapter['title'], degree=1)
# Add chapter content material as regular textual content
doc.add_paragraph(chapter['content'])
# Add a web page break after every chapter
doc.add_page_break()
# Save the doc
doc.save(output_filename)
Whereas utilizing GPT and APIs for translation is quick and environment friendly, there are key limitations in comparison with human translation:
- Pronoun and Reference Errors: GPT did misread pronouns or references in few instances, probably attributing actions or statements to the fallacious particular person within the narrative. A human translator can higher resolve such ambiguities.
- Cultural Context: GPT missed delicate cultural references and idioms {that a} human translator might interpret extra precisely. On this case, a number of slang phrases distinctive to Santa Cruz, Bolivia, have been retained within the unique language with out extra context or rationalization.
Combining AI with human assessment can steadiness pace and high quality, making certain translations are each correct and genuine.
This undertaking demonstrates an method to translating a guide utilizing a mix of GPT-3 and Unstructured APIs. By automating the interpretation course of, we considerably lowered the guide effort required. Whereas the preliminary translation output might require some minor human revisions to refine the nuances and make sure the highest high quality, this method serves as a powerful basis for environment friendly and efficient guide translation
If in case you have any suggestions or solutions on the way to enhance this course of or the standard of the translations, please be happy to share them within the feedback beneath.