• User needs to generate code for transforming datasets for SageMaker model training or model evaluation.
• A dataset requires processing, cleaning, or formatting before training or evaluation.
• Workflow requires a formal review and approval cycle before execution.

1. One thing at a time. Each response advances exactly one decision. Never combine multiple questions or recommendations in a single turn.
2. Confirm before proceeding. Wait for the user to agree before moving to the next step. You are a guide, not a runaway train.
3. Don't read files until you need them. Only read reference files when you've reached the workflow step that requires them and the user has confirmed the direction. Never read ahead.
4. No narration. Don't explain what you're about to do or what you just did. Share outcomes and ask questions. Keep responses short and focused.
5. No repetition. If you said something before a tool call, don't repeat it after. Only share new information.
6. Do not deviate from the Workflow. The steps listed in the workflow should be followed exactly as described. Progress from Step 1 to Step 10 to complete the task. Do not deviate from the workflow!
7. Always end with a question. Whenever you pause for user input, acknowledgment, or feedback, your response must end with a question. Never leave the user with a statement and expect them to know they need to respond.
8. Never overwrite existing files — append instead. If a target notebook already exists, do NOT overwrite it. Append new cells to the existing file. Notify the user that the file already exists and that you will be appending to it.
9. Avoid filename collisions. When creating a new file, check if a file with the same name already exists. If it does, rename the new file by appending a numeric suffix (e.g.,

   transform_dataset_2.ipynb

   ) before writing.
10. Default output format is JSONL. Unless the user explicitly requests a different file format, the transformed dataset should be written as

    .jsonl

    (JSON Lines — one JSON object per line).

1. Try fetching the live documentation at https://docs.aws.amazon.com/sagemaker/latest/dg/model-customize-evaluation-dataset-formats.html to get the latest evaluation dataset schema definitions.
2. If the fetch fails (e.g., no internet access, VPC environment), fall back to the offline copy at

   references/sagemaker_dataset_formats.md

   . Inform the user that the format schemas are from an offline copy and may be outdated.

"Is this dataset transformation for model training or model evaluation? This helps me look up the right target format for you."

• Training → format resolution will use the local training data requirements reference (model type + finetuning technique dependent).
• Evaluation → format resolution will use the live AWS documentation (with offline fallback).

"I can help you transform your dataset's format! Here's my plan: I will first need to understand the format of your dataset and the transformation requirements. Once I have that, I will generate a dataset transformation function that we can refine together. After the dataset transformation function is refined to your liking, I will perform the transformation task and upload it to your desired location! Does this sound good?"

"What's the dataset format you would like to transform it into?"

• If training data: Ask the user for the finetuning technique (SFT, DPO, RLVR) and model type (Open Weights like Llama/Qwen vs Nova) if not already known. Then look up the required format from the "Training Data Formats" section in the Known Dataset Formats Reference above.
• If evaluation data: If the user mentions a well-known format name (e.g., "OpenAI format", "SageMaker format"), fetch the schema from the live documentation as described in the "Evaluation Data Formats" section above. If a well-known format is fetched, confirm with the user:

"I've found a SageMaker dataset format: {sagemaker-dataset-format-name} with schema: {sagemaker-dataset-format-schema}. Is this what you were referring to?"

"Where can I find your dataset? Either a local directory or S3 location works!"

"Where should I output your transformed dataset to? Either a local directory or S3 location works!"

{original_name}_{target_format}.jsonl

gen_qa_100k_openai.jsonl

references/dataset_transformation_code.md

def transform_dataset(df: pd.DataFrame) -> pd.DataFrame:

%%writefile <project-dir>/scripts/transform_fn.py

<project-dir>

dpo-to-rlvr-conversion

<project-dir>/notebooks/

<project-dir>/scripts/

1. Generate the transformation function.
2. Write the sample data to a temporary JSONL file (e.g.,

   /tmp/test_input.jsonl

   ), then run:

   python3 -c "import sys; sys.path.insert(0, '<project-dir>/scripts'); from transform_fn import transform_dataset; import pandas as pd; df = pd.read_json('/tmp/test_input.jsonl', lines=True); result = transform_dataset(df); print(result.to_json(orient='records', lines=True))"

3. If the test fails, fix and re-test until it passes.
4. Show the user the function and transformed sample output for review.

• references/notebook_structure.md

  (cell order, placeholders, and content)

• references/notebook_writing_guide.md

  (Jupyter notebook JSON formatting)

• Add a

  %%writefile <project-dir>/scripts/<script_name>.py

  code cell to the notebook AND write the file to disk for testing.
• The script must import

  transform_dataset

  from

  transform_fn

  .
• Replace placeholders with the actual input/output paths.

references/dataset_transformation_code.md

1. Write the sample records to a temporary JSONL file (e.g.,

   /tmp/test_input.jsonl

   ).
2. Run:

   python3 <project-dir>/scripts/<script_name> --input /tmp/test_input.jsonl --output /tmp/test_output.jsonl

3. If it fails, debug and fix, then re-run until successful.
4. Show the user the output for review.

• If the dataset is in S3, use the AWS MCP tool

  head-object

  (S3 service) with the bucket and key to get

  ContentLength

  .
• If the dataset is local, check the file size.

• Dataset < 50 MB → recommend local execution
• Dataset ≥ 50 MB → recommend SageMaker Processing Job

"Your dataset is {size} MB — since it's under 50 MB, I'd recommend running the transformation locally. Would you like to proceed with local execution, or would you prefer a SageMaker Processing Job instead?"

"Your dataset is {size} MB — since it's over 50 MB, I'd recommend running this as a SageMaker Processing Job for better performance. Would you like to proceed with a SageMaker Processing Job, or would you prefer to run it locally instead?"

• Add a cell that runs the transformation by importing from the

  .py

  files already on disk (written by the agent during Steps 7–8): import

  transform_dataset

  from

  transform_fn

  , load the dataset, transform, and save output. Scripts are located in

  <project-dir>/scripts/

  .

• Add a cell that submits and monitors the Processing Job inline using the V3 SageMaker SDK directly (FrameworkProcessor, ProcessingInput, ProcessingOutput, etc.). Create a FrameworkProcessor with the SKLearn 1.2-1 image, configure inputs/outputs, and call

  processor.run(wait=True, logs=True)

  to block the cell and stream logs until the job completes. See

  scripts/transformation_tools.py

  for reference implementation details.
• Inform the user they can run this cell to kick off and monitor the job.

"I've added the execution cell to the notebook. You can run it to transform the full dataset. Would you like to review the notebook before running it?"

• Read 1–2 sample records from the output to show the user.
• Report the total number of records transformed.
• Ask the user if the output looks good.