Usually, an object detection mannequin is skilled with a hard and fast vocabulary, that means it will possibly solely acknowledge a predefined set of object classes. Nevertheless, in our pipeline, since we are able to’t predict upfront which objects will seem within the picture, we’d like an object detection mannequin that’s versatile and able to recognizing a variety of object courses. To attain this, I take advantage of the OWL-ViT mannequin [11], an open-vocabulary object detection mannequin. This mannequin requires textual content prompts that specifies the objects to be detected.
One other problem that must be addressed is acquiring a high-level concept of the objects current within the picture earlier than using the OWL-ViT mannequin, because it requires a textual content immediate describing the objects. That is the place VLMs come to the rescue! First, we move the picture to the VLM with a immediate to determine the high-level objects within the picture. These detected objects are then used as textual content prompts, together with the picture, for the OWL-ViT mannequin to generate detections. Subsequent, we plot the detections as bounding bins on the identical picture and move this up to date picture to the VLM, prompting it to generate a caption. The code for inference is partially tailored from [12].
# Load mannequin immediately
from transformers import AutoProcessor, AutoModelForZeroShotObjectDetectionprocessor = AutoProcessor.from_pretrained("google/owlvit-base-patch32")
mannequin = AutoModelForZeroShotObjectDetection.from_pretrained("google/owlvit-base-patch32")
I detect the objects current in every picture utilizing the VLM:
IMAGE_QUALITY = "excessive"
system_prompt_object_detection = """You're supplied with a picture. You should determine all essential objects within the picture, and supply a standardized checklist of objects within the picture.
Return your output as follows:
Output: object_1, object_2"""user_prompt = "Extract the objects from the offered picture:"
detected_objects = process_images_in_parallel(image_paths, system_prompt=system_prompt_object_detection, user_prompt=user_prompt, mannequin = "gpt-4o-mini", few_shot_prompt= None, element=IMAGE_QUALITY, max_workers=5)
detected_objects_cleaned = {}for key, worth in detected_objects.gadgets():
detected_objects_cleaned[key] = checklist(set([x.strip() for x in value.replace("Output: ", "").split(",")]))
The detected objects at the moment are handed as textual content prompts to the OWL-ViT mannequin to acquire the predictions for the pictures. I implement a helper operate that predicts the bounding bins for the pictures, after which plots the bounding field on the unique picture.
from PIL import Picture, ImageDraw, ImageFont
import numpy as np
import torchdef detect_and_draw_bounding_boxes(
image_path,
text_queries,
mannequin,
processor,
output_path,
score_threshold=0.2
):
"""
Detect objects in a picture and draw bounding bins over the unique picture utilizing PIL.
Parameters:
- image_path (str): Path to the picture file.
- text_queries (checklist of str): Record of textual content queries to course of.
- mannequin: Pretrained mannequin to make use of for detection.
- processor: Processor to preprocess picture and textual content queries.
- output_path (str): Path to save lots of the output picture with bounding bins.
- score_threshold (float): Threshold to filter out low-confidence predictions.
Returns:
- output_image_pil: A PIL Picture object with bounding bins and labels drawn.
"""
img = Picture.open(image_path).convert("RGB")
orig_w, orig_h = img.measurement # authentic width, peak
inputs = processor(
textual content=text_queries,
photographs=img,
return_tensors="pt",
padding=True,
truncation=True
).to("cpu")
mannequin.eval()
with torch.no_grad():
outputs = mannequin(**inputs)
logits = torch.max(outputs["logits"][0], dim=-1) # form (num_boxes,)
scores = torch.sigmoid(logits.values).cpu().numpy() # convert to chances
labels = logits.indices.cpu().numpy() # class indices
boxes_norm = outputs["pred_boxes"][0].cpu().numpy() # form (num_boxes, 4)
converted_boxes = []
for field in boxes_norm:
cx, cy, w, h = field
cx_abs = cx * orig_w
cy_abs = cy * orig_h
w_abs = w * orig_w
h_abs = h * orig_h
x1 = cx_abs - w_abs / 2.0
y1 = cy_abs - h_abs / 2.0
x2 = cx_abs + w_abs / 2.0
y2 = cy_abs + h_abs / 2.0
converted_boxes.append((x1, y1, x2, y2))
draw = ImageDraw.Draw(img)
for rating, (x1, y1, x2, y2), label_idx in zip(scores, converted_boxes, labels):
if rating < score_threshold:
proceed
draw.rectangle([x1, y1, x2, y2], define="crimson", width=3)
label_text = text_queries[label_idx].change("A picture of ", "")
text_str = f"{label_text}: {rating:.2f}"
text_size = draw.textsize(text_str) # If no font used, take away "font=font"
text_x, text_y = x1, max(0, y1 - text_size[1]) # place textual content barely above field
draw.rectangle(
[text_x, text_y, text_x + text_size[0], text_y + text_size[1]],
fill="white"
)
draw.textual content((text_x, text_y), text_str, fill="crimson") # , font=font)
img.save(output_path, "JPEG")
return img
for key, worth in tqdm(detected_objects_cleaned.gadgets()):
worth = ["An image of " + x for x in value]
detect_and_draw_bounding_boxes(key, worth, mannequin, processor, "images_with_bounding_boxes/" + key.break up("/")[-1], score_threshold=0.15)
The pictures with the detected objects plotted at the moment are handed to the VLM for captioning:
IMAGE_QUALITY = "excessive"
image_paths_obj_detected_guided = [x.replace("downloaded_images", "images_with_bounding_boxes") for x in image_paths] system_prompt="""You're a useful assistant that may analyze photographs and supply captions. You're supplied with photographs that additionally include bounding field annotations of the essential objects in them, together with their labels.
Analyze the general picture and the offered bounding field info and supply an acceptable caption for the picture.""",
user_prompt="Please analyze the next picture:",
obj_det_zero_shot_high_quality_captions = process_images_in_parallel(image_paths_obj_detected_guided, mannequin = "gpt-4o-mini", few_shot_prompt= None, element=IMAGE_QUALITY, max_workers=5)
On this process, given the easy nature of the pictures we use, the placement of the objects doesn’t add any vital info to the VLM. Nevertheless, Object Detection Guided Prompting could be a highly effective device for extra advanced duties, equivalent to Doc Understanding, the place format info may be successfully offered by way of object detection to the VLM for additional processing. Moreover, Semantic Segmentation may be employed as a way to information prompting by offering segmentation masks to the VLM.
VLMs are a robust device within the arsenal of AI engineers and scientists for fixing a wide range of issues that require a mixture of imaginative and prescient and textual content abilities. On this article, I discover prompting methods within the context of VLMs to successfully use these fashions for duties equivalent to picture captioning. That is in no way an exhaustive or complete checklist of prompting methods. One factor that has turn into more and more clear with the developments in GenAI is the limitless potential for inventive and revolutionary approaches to immediate and information LLMs and VLMs in fixing duties.
[1] J. Chen, H. Guo, Ok. Yi, B. Li and M. Elhoseiny, “VisualGPT: Knowledge-efficient Adaptation of Pretrained Language Fashions for Picture Captioning,” 2022 IEEE/CVF Convention on Pc Imaginative and prescient and Sample Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 18009–18019, doi: 10.1109/CVPR52688.2022.01750.
[2] Luo, Z., Xi, Y., Zhang, R., & Ma, J. (2022). A Frustratingly Easy Method for Finish-to-Finish Picture Captioning.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millicah, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. 2022. Flamingo: a visible language mannequin for few-shot studying. In Proceedings of the thirty sixth Worldwide Convention on Neural Data Processing Techniques (NIPS ‘22). Curran Associates Inc., Crimson Hook, NY, USA, Article 1723, 23716–23736.
[4] https://huggingface.co/blog/vision_language_pretraining
[5] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Proceedings of the 56th Annual Assembly of the Affiliation for Computational Linguistics (Quantity 1: Lengthy Papers), pages 2556–2565, Melbourne, Australia. Affiliation for Computational Linguistics.
[6] https://platform.openai.com/docs/guides/vision
[7] Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Textual content Summarization Branches Out, pages 74–81, Barcelona, Spain. Affiliation for Computational Linguistics.
[8]https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/traditional/#rouge-score
[9] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., … & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in giant language fashions. Advances in Neural Data Processing Techniques, 35, 24824–24837.
[11] Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. 2022. Easy Open-Vocabulary Object Detection. In Pc Imaginative and prescient — ECCV 2022: seventeenth European Convention, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Half X. Springer-Verlag, Berlin, Heidelberg, 728–755. https://doi.org/10.1007/978-3-031-20080-9_42
[13] https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/gpt-4-v-prompt-engineering