BigFrames Development Standards

• Avoid

  .to_pandas()

  : You MUST NOT use

  .to_pandas()

  to download the entire dataset into memory as this downloads all data to the client's memory, bypassing BigQuery's distributed computation and risking Out of Memory (OOM) errors. There are some exceptions:
  • An error message explicitly requests you to use

    to_pandas()

  • You are going to visualize the data, and the visualization library does not accept BigFrames Dataframe/Series instances. In this case, reduce the amount of data you are going to download before calling

    .to_pandas()

• Avoid

  read_gbq()

  for SQL: Do not write SQL queries and execute them with

  read_gbq()

  to maintain the Pandas-like DataFrame abstraction and allow lazy executions. Use BigFrames Dataframe/Series methods instead.
• Use BigFrames ML package for Machine Learning Tasks: Do not use Scikit-learn or other ML libraries with BigFrames dataframes because standard Scikit-learn models require bringing data into local client memory, whereas bigframes.ml delegates training directly to BigQuery's scalable ML engine. Import your tools/classes from

  bigframes.ml

  .
• Stay in the Cloud: Perform data cleaning, transformation, and analysis via BigFrames methods to leverage BigQuery's scale.
• Accessors over UDFs/Lambdas:
  • Prefer built-in accessors (e.g.,

    df.col.str.*

    ,

    df.col.dt.*

    ) over remote UDFs.
  • Do not use lambdas with

    Series.map()

    or

    DataFrame.apply()

    .
• Schema Verification: Do not assume schema of intermediate outputs. Check

  .dtypes

  after loading, and use

  display()

  with

  .head()

  or

  .peek()

  .
• Visualization: BigFrames Dataframe mostly works directly with Matplotlib, Seaborn, and other plotting libraries. If your attempt didn't work, try using the

  plot

  accessor. If that didn't work either, you MUST sample or aggregate your data to make it small enough before calling

  to_pandas()

  .

Model Development

• Unlike Scikit-learn: BigFrames'

  predict()

  method always returns a DataFrame containing both predictions and features (not just a series of predictions).
• No

  random_state

  : Do not pass a

  random_state

  argument when instantiating BigFrames ML models, because this parameter is not supported in the BigFrames ML package.
• Automatic Scaling: Do not use

  OneHotEncoder

  or

  StandardScaler

  unless explicitly requested (handled automatically).
• Hyperparameter Tuning: You must write custom loops (BigFrames lacks

  GridSearchCV

  or

  RandomizedSearchCV

  ).
• ARIMA Plus (Forecasting):
  • Import from

    bigframes.ml.forecasting

    .
  • Sort data chronologically and split around a timepoint before training.
  • Prediction horizon must be less than or equal to training horizon.
• PCA: BigFrames' PCA class lacks simple

  transform()

  method. Use

  predict()

  instead.
• Model Persistence: To persist a model, use

  model.to_gbq()

  . To load a persisted model, use

  bpd.read_gbq_model()

  .