Recently, I have come across a youtube video that is quite inspiring. It explains what AI Engineers do and how to develop AI applications in PostgreSQL. It made me believe I can develop something useful as if I am an AI Engineer! At least I know Python and PostgreSQL reasonably well.
One of the functionalities of the DBAETS application I developed is it is a knowledge base for DBAs. From time to time, I post notes there about database related activities, such as troubleshooting, performance tuning and so on and so forth. My ambition is to develop a Q&A RAG application for DBAs to ask questions and get answers utilizing the knowledge-base data in the DBAETS.
What is RAG?
"In the field of AI, RAG stands for Retrieval-Augmented Generation. It's a framework that combines the strengths of traditional information retrieval systems with generative large language models (LLMs). RAG enhances AI responses by retrieving relevant information from external sources, such as databases or web pages, and using that information to augment the LLM's generation process. This approach helps create more accurate, up-to-date, and relevant text outputs. "
Let the journey begin! And I will post any meaningful progress along the way.
As of today, I have achieved three things as described in the following:
- I turned my PostgreSQL db as a vector database by installing the pgvector extension
– installation of pgvector
pgdbhost003.mycompany.com, as root
cd /u01/stage
git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
cd pgvector
make
make install
– enable the extension
postgres=# \c etsdb
You are now connected to database "etsdb" as user "postgres".
etsdb=# CREATE EXTENSION vector;
CREATE EXTENSION
etsdb=#
– create a table with `vector` data type
etsdb=# create table items(id bigserial primary key, embedding vector(3));
CREATE TABLE
– insert
etsdb=# INSERT INTO items (embedding) VALUES ('[1,2,3]'), ('[4,5,6]');
INSERT 0 2
– query
etsdb=# SELECT * FROM items ORDER BY embedding <-> '[3,1,2]' LIMIT 5;
id | embedding
----+-----------
1 | [1,2,3]
2 | [4,5,6]
As you can see I have successfully turned my PostgreSQL database into a vector store.
- I am able to call Gemini API
In the following example script, I ask a LLM model what an AI Engineer do through an API
(venv311_prac) C:\Users\dsun\Denis_files\python_proj\ragtest\ragtest>type test_googleapi.py
# ref : https://github.com/google-gemini/cookbook
# ref : https://github.com/googleapis/python-genai
#
# RAG:
#
# https://medium.com/@saurabhgssingh/understanding-rag-building-a-rag-system-from-scratch-with-gemini-api-b11ad9fc1bf7
#
#
# type of services
# ** Gemini Developer API
# ** Vertex AI
# pip install google-genai
#
import os
from google import genai
from google.genai import types
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))
response = client.models.generate_content(
model="gemini-2.0-flash-001",
contents="Explain what an AI Enigeer do?",
)
print(response.text)
Run the program:
(venv311_prac) C:\Users\dsun\Denis_files\python_proj\ragtest\ragtest>python test_googleapi.py
An AI Engineer is a professional who focuses on **applying artificial intelligence (AI) and machine learning (ML) models to solve real-world problems.** They bridge the gap between theoretical AI research and practical implementation. Think of them as the builders and architects of AI systems.
Here's a breakdown of what an AI Engineer typically does:
**1. Data Handling and Preparation:**
* **Data Collection:** Gathering data from various sources, including databases, APIs, and other data stores.
* **Data Cleaning and Preprocessing:** Cleaning messy data (handling missing values, outliers, and inconsistencies) to make it suitable for training ML models. This is a crucial step as the quality of the data directly impacts the model's performance.
* **Data Transformation:** Converting data into a format that can be easily processed by ML algorithms. This might involve feature scaling, encoding categorical variables, and creating new features.
* **Data Augmentation:** Generating synthetic data to increase the size and diversity of the training dataset, which can improve model robustness.
**2. Model Development and Training:**
* **Model Selection:** Choosing the appropriate AI/ML model based on the problem, data characteristics, and desired outcome (e.g., regression, classification, clustering, or a specific deep learning architecture).
* **Model Training:** Using the prepared data to train the selected model. This involves tuning the model's parameters to optimize its performance.
* **Model Evaluation:** Assessing the trained model's performance using various metrics and techniques (e.g., accuracy, precision, recall, F1-score, ROC curves).
* **Hyperparameter Tuning:** Optimizing the model's hyperparameters (parameters that are not learned during training) to improve its performance. This often involves using techniques like grid search or Bayesian optimization.
**3. Model Deployment and Maintenance:**
* **Model Deployment:** Putting the trained model into a production environment where it can be used to make predictions on new data. This could involve deploying the model to a cloud platform, a server, or an embedded device.
* **Model Monitoring:** Tracking the model's performance in production to ensure it continues to perform as expected.
* **Model Retraining:** Retraining the model with new data to keep it up-to-date and prevent performance degradation (model drift).
* **Infrastructure Management:** Managing the infrastructure required to run the AI system, including servers, databases, and cloud resources.
**4. Collaboration and Communication:**
* **Collaboration:** Working closely with data scientists, software engineers, and other stakeholders to build and deploy AI solutions.
* **Communication:** Effectively communicating technical concepts and findings to both technical and non-technical audiences.
* **Documentation:** Creating and maintaining documentation for the AI system, including the model architecture, training process, and deployment instructions.
**Key Skills for AI Engineers:**
* **Programming:** Strong proficiency in programming languages like Python (most common), Java, or C++.
* **Machine Learning:** A solid understanding of machine learning algorithms and techniques.
* **Deep Learning:** Experience with deep learning frameworks like TensorFlow, PyTorch, or Keras.
* **Data Science Fundamentals:** Knowledge of data analysis, data visualization, and statistical modeling.
* **Data Engineering:** Skills in data wrangling, data cleaning, and data pipelining.
* **Cloud Computing:** Experience with cloud platforms like AWS, Azure, or Google Cloud.
* **DevOps:** Familiarity with DevOps practices like continuous integration and continuous delivery (CI/CD).
* **Software Engineering:** Understanding of software development principles, design patterns, and testing methodologies.
* **Problem-Solving:** Strong analytical and problem-solving skills.
* **Communication:** Excellent communication and collaboration skills.
**In summary, an AI Engineer is a hands-on role that requires a blend of programming, machine learning, and software engineering skills. They are responsible for building, deploying, and maintaining AI systems that solve real-world problems.** They are not just theorists; they are builders who bring AI models to life.
- I can generate embeddings:
(venv311_prac) C:\Users\dsun\Denis_files\python_proj\ragtest\ragtest>type gen_emb.py
from google import genai
import os
client = genai.Client(api_key=os.getenv("GEMINI_API_KEY"))
result = client.models.embed_content(
model="gemini-embedding-exp-03-07",
contents="What is the meaning of life?")
print(result.embeddings)
(venv311_prac) C:\Users\dsun\Denis_files\python_proj\ragtest\ragtest>python gen_emb.py
[ContentEmbedding(values=[-0.022372285, -0.004451784, 0.013473644, -0.053762246, -0.020569915, 0.011864573, 0.015185799, 0.006950965, 0.03180835, 0.007074574, 0.027503368, -0.00600613, -0.014889315, 0.03269886, 0.12054204, 0.019322146, 0.000517173, 0.0045754807, -0.00856155, -0.01532448, 0.015616342, -0.008661197, -0.017454486, 0.0099245, -0.015551475, 0.012284064, 0.020809751, -0.0037114064, 0.025106275, 0.008105811, 0.020252233, 0.0019548477, -0.010780675, 0.027334962, -0.017213175, -0.011735542, 0.009507163, -0.015499457, -0.013591795, 0.0138707, -0.022853972, -0.009638755, -0.0034423112, -0.018855078, 0.018475225, -0.010515843, 0.015031793, -0.042978574, -0.013993226, 0.007916359, -0.012274015, 0.011758872, -0.010251529, -0.15881006, 0.016281825, 0.0103672175, -0.006364179, -0.009997806, -0.025991082, -0.027687864, -0.008586312, -0.014933809, -0.007584574, -0.021427568, 0.008805106, -0.009369353, -0.02012641, 0.011695516, 0.0020037016, 0.012606018, -0.01629937, 0.015133599, -0.005364407, -0.013935832, 0.0065723164, 0.014414399, 0.016648233, -0.009122886, 0.0006896179, -0.0042646583, 0.0001541545, -0.01034019, -0.025387881, -0.01852225, -0.005335359, 0.013288918, -0.008168338, -0.005782173, 0.017077502, -0.0015941358, 0.027414756, 0.00015515718, 0.019269329, 0.026197737, -0.015433854, -0.016303977, -0.0048079626, -0.013850489, 0.018751703, 0.0037166988, -0.029702129, -0.018824589, 0.0037240938, -0.00461113, 0.0010804693, 0.0029576228, 0.020787966, 0.014922842, 0.010893234, -0.00839888, 0.016467841, -0.011249123, -0.014664661, -9.935649e-06, -0.016037786, -0.17255014, 0.011858983, -0.0148698045, -0.019271476, 0.02731455, 0.029812986, 0.0046130605, -0.012905329, 0.010795695, -0.0144493915, 0.0009820901, 0.02241333 ..... ]
In summary, I have successfully enabled vector capabilities in a PostgreSQL database and demonstrated the ability to call the Gemini API. This blog post serves as a starting point of my exploration toward developing practical AI solutions.
