Skip to main content

datalinks package

Subpackages

Submodules

Bases: object Command-Line Interface (CLI) wrapper for customizable argument parsing. Simplifies the creation and usage of the DataLinks CLI by allowing to pass a custom callback function for additional arguments specific to an application. It provides a standard set of CLI arguments while enabling customization through user-defined groups.
  • Variables:
    • name – The name of the CLI program.
    • description – The description of the CLI program.
  • Return type: Namespace
Bases: object The base type for entity resolution operators.
  • Variables: targetColumns – A list of column names that are the target for matching. If None, all columns are used for entity resolution.

targetColumns : List[str] | None = None

Bases: MatchType Use this match type to evaluate and configure specific exact matching criteria for the data values.
  • Variables:
    • minVariation – Minimum allowable variation in the field to check for matches (defaults to 0.0).
    • minDistinct – Minimum percentage of distinct values in the field to check for matches (defaults to 0.0).

minVariation : float | None = None

minDistinct : float | None = None

Bases: MatchType use this match type to check for matches in fields that represent geographical attributes.
  • Variables:
    • distance – The maximum distance value for the geographical match (defaults to 2.0)
    • distanceUnit – The unit of measurement for the distance, such as kilometers or miles (defaults to ‘kilometers’).

distance : float | None = None

distanceUnit : str | None = None

Bases: StrEnum Enumerates the various resolution strategies for handling matching or reconciliation of entity data. Each enumeration value specifies a particular method or approach used for determining entity equivalence or correspondence.
  • Variables:
    • ExactMatch – Used when entities are determined to be equivalent based on exact value matches without any approximation.
    • GeoMatch – Used when entities are matched based on their geographical location or proximity.

ExactMatch = ‘ExactMatch’

GeoMatch = ‘GeoMatch’

Bases: object Encapsulates configuration related to different types of entity resolution matches. This class is designed to store, manage, and provide access to various entity resolution match type configurations, such as ExactMatch and GeoMatch. It maintains internal state for these match types and also provides access to a consolidated configuration in dictionary format.
  • Variables: matchTypes – A dictionary mapping entity resolution types to their respective match configurations (e.g., ExactMatch, GeoMatch).

matchTypes : dict[EntityResolutionTypes, MatchType | None]

property config : Dict[str, Dict] | None

Bases: ABC Abstract base class for loading resources from a specified folder. It serves as a template for loading files or other resources while maintaining consistency across different implementations.
  • Variables: folder – Path to the folder from which resources will be loaded.

abstractmethod load_from_folder()

Bases: Loader A loader for processing JSON files in a specified folder. Iterates through all .json files within a given folder, parses their content, and processes each JSON object into a standardized format using the load_item method.
  • Variables: folder – Path to the folder containing JSON files. All .json files in this folder will be processed.

load_from_folder()

  • Return type: List[Dict[str, str]]

abstractmethod load_item(row)

  • Return type: Dict[str, str]
Request and response models, re-exported for type annotations. Generated by make codegen from spec/openapi.yaml. Do not edit. Bases: ResponseModel

session_id : Annotated[UploadSessionId, FieldInfo(annotation=NoneType, required=True, alias=‘sessionId’, alias_priority=2)]

session_status : Annotated[SessionStatus, FieldInfo(annotation=NoneType, required=True, alias=‘sessionStatus’, alias_priority=2, description=‘Status of the upload session.’, examples=[‘preparing’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

upload_id : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘uploadId’, alias_priority=2, description=‘Upload ID to abort.’, examples=[‘2~VmxqYXRlRDZWfjJrRWRiZ2FEQXk3dHI4DGhGLygtTmQNXEZPQXFzMVZOZTUgdEdrXCEyNStkQUp3U3Q2RjJqSkV4MFNBOWxJCk0yIHEkfcQ’])]

key : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘S3 object key to abort.’, examples=[‘uploads/data/20240301-143022-uuid.json’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

username : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Username of the user that has access.’, examples=[‘alice’])]

role : Annotated[Role, FieldInfo(annotation=NoneType, required=True, description=‘Access level granted to the user.’, examples=[‘editor’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

from_ : ApiStringColumnIdentifier, FieldInfo(annotation=NoneType, required=True, alias=‘from’, alias_priority=2, examples=[{‘username’: ‘johndoe’, ‘namespace’: ‘cinematography’, ‘dataset’: ‘movies’, ‘columnName’: ‘director’}])]

to : ApiStringColumnIdentifier, FieldInfo(annotation=NoneType, required=True, examples=[{‘username’: ‘janedoe’, ‘namespace’: ‘entertainment’, ‘dataset’: ‘films’, ‘columnName’: ‘director_name’}])]

options : Annotated[ExactMatchConfig | GeoMatchConfig | None, FieldInfo(annotation=NoneType, required=True, description=‘Flexible options map for match configuration.’)]

match_type : Annotated[MatchType, FieldInfo(annotation=NoneType, required=True, alias=‘matchType’, alias_priority=2, description=‘Type of match to create between the columns.’, examples=[‘ExactMatch’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

thumbs_up : Annotated[bool, FieldInfo(annotation=NoneType, required=True, alias=‘thumbsUp’, alias_priority=2, description=‘True for a thumbs up, false for a thumbs down.’)]

comment : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘Optional free-text comment accompanying the rating.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

run_id : Annotated[UUID, FieldInfo(annotation=NoneType, required=True, alias=‘runId’, alias_priority=2)]

agent_kind : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘agentKind’, alias_priority=2, examples=[‘auto_rag’])]

question : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The original prompt that started the conversation.’)]

status : Status

can_resume : Annotated[bool, FieldInfo(annotation=NoneType, required=True, alias=‘canResume’, alias_priority=2, description=‘True only when the status is running AND the checkpoint blob loads cleanly.’)]

last_node_id : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘lastNodeId’, alias_priority=2)]

error_message : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘errorMessage’, alias_priority=2)]

result : Annotated[dict[str, Any] | None, FieldInfo(annotation=NoneType, required=True, description=“Rendered AutoRag result for completed runs, in the same shape as POST /query/autorag’s response body.”)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

name : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Name of the dataset.’)]

namespace : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Name of the namespace the dataset belongs to.’)]

name_with_namespace : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘nameWithNamespace’, alias_priority=2, description=‘The name concatenated with the namespace.’, deprecated=True)]

Read-only data descriptor used to emit a runtime deprecation warning before accessing a deprecated field.

msg

The deprecation message to be emitted.

wrapped_property

The property instance if the deprecated field is a computed field, or None.

field_name

The name of the field being deprecated.

owner : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The username associated with the data.’)]

user : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The username associated with the data.’, deprecated=True)]

Read-only data descriptor used to emit a runtime deprecation warning before accessing a deprecated field.

msg

The deprecation message to be emitted.

wrapped_property

The property instance if the deprecated field is a computed field, or None.

field_name

The name of the field being deprecated.

visibility : Annotated[Visibility, FieldInfo(annotation=NoneType, required=True, description=‘Indicates whether the data is public or private.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

column_names : Annotated[list[str], FieldInfo(annotation=NoneType, required=True, alias=‘columnNames’, alias_priority=2, description=‘List of dataset column names.’, examples=[[‘id’, ‘name’, ‘age’, ‘department’, ‘salary’]])]

data_description : Annotated[DataDescription | None, FieldInfo(annotation=NoneType, required=True, alias=‘dataDescription’, alias_priority=2)]

schema_definition : Annotated[SchemaDefinition | None, FieldInfo(annotation=NoneType, required=True, alias=‘schemaDefinition’, alias_priority=2)]

row_count : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘rowCount’, alias_priority=2, description=‘Number of rows in dataset.’, examples=[1250])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

destination : Annotated[str, FieldInfo(annotation=NoneType, required=True, examples=[‘ri.user.examples.companies.other_addresses’])]

hits : Annotated[int, FieldInfo(annotation=NoneType, required=True, examples=[5])]

match_type : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘matchType’, alias_priority=2, examples=[‘ExactMatch’])]

origin : Annotated[str, FieldInfo(annotation=NoneType, required=True, examples=[‘ri.user.examples.companies.nationalities’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

name : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The name of the namespace.’, examples=[‘cinematography’])]

owner : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Username of the namespace owner.’, examples=[‘datalinks’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

username : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Username of the dataset owner.’, examples=[‘johndoe’])]

namespace : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Namespace of the dataset.’, examples=[‘cinematography’])]

dataset : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Name of the dataset.’, examples=[‘movies’])]

column_name : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘columnName’, alias_priority=2, description=‘Column name in the dataset.’, examples=[‘director’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

response : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=“The final synthesized answer to the user’s question.”)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

message : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Error message describing what went wrong.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

steps : Annotated[list[str], FieldInfo(annotation=NoneType, required=True, description=‘List of step descriptions that will be executed.’)]

reasoning : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The reasoning produced by the LLM when generating the plan.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

index : Annotated[int, FieldInfo(annotation=NoneType, required=True, description=‘Zero-based index of the step whose query is executing.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

username : str

namespace : str

query : Annotated[str, FieldInfo(annotation=NoneType, required=True, examples=[‘What do llamas drink to cool themselves down?’])]

helper_prompt : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘helperPrompt’, alias_priority=2, examples=[“You’re a prophet of the alpacalypse. Your task is to spit the word!”])]

model : str | None

provider : str | None

models : AutoRagStageModels | None

conversation_id : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘conversationId’, alias_priority=2, description=‘Existing conversation to continue. Omit to start a new conversation; the id is returned on the run-started SSE event.’)]

web_search : Annotated[bool | None, FieldInfo(annotation=NoneType, required=True, alias=‘webSearch’, alias_priority=2, description=‘Override web search for this request: false disables it; omit (or true) to use the deployment default.’)]

caller_context : Annotated[CallerContext | None, FieldInfo(annotation=NoneType, required=True, alias=‘callerContext’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

response : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=“Final response generated for the user’s question.”)]

steps : Annotated[list[Step], FieldInfo(annotation=NoneType, required=True, description=‘List of steps executed during the AutoRAG process.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

model : str | None

provider : str | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

plan : AutoRagStageModel | None

query : AutoRagStageModel | None

answer : AutoRagStageModel | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

index : Annotated[int, FieldInfo(annotation=NoneType, required=True, description=‘Zero-based index of this step in the plan.’)]

instruction : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The instruction generated and executed for this step.’)]

query : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The query derived from the instruction.’)]

reasoning : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The reasoning produced by the LLM when generating the query for this step.’)]

data : DataArray

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

index : Annotated[int, FieldInfo(annotation=NoneType, required=True, description=‘Zero-based index of the step that just started.’)]

instruction : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The plan instruction for this step.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootModel[Union[AutoRagPlanEvent, AutoRagStepEvent, AutoRagAnswerEvent, AutoRagErrorEvent, AutoRagPlanningStartedEvent, AutoRagStepStartedEvent, AutoRagQueryExecutingEvent, AutoRagAnswerStartedEvent]]

root : AutoRagPlanEvent | AutoRagStepEvent | AutoRagAnswerEvent | AutoRagErrorEvent | AutoRagPlanningStartedEvent | AutoRagStepStartedEvent | AutoRagQueryExecutingEvent | AutoRagAnswerStartedEvent

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

turns : Annotated[list[CallerContextTurn] | None, FieldInfo(annotation=NoneType, required=True, description=‘Prior exchange in the calling system, oldest first.’)]

facts : Annotated[list[CallerContextFact] | None, FieldInfo(annotation=NoneType, required=True, description=‘Values the calling system asserts. Where a label repeats, the later entry supersedes the earlier one.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

label : Annotated[str, FieldInfo(annotation=NoneType, required=True, examples=[‘tenant’])]

value : Annotated[str, FieldInfo(annotation=NoneType, required=True, examples=[‘acme’])]

source : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘Which upstream agent asserted this fact.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

role : Role2

text : str

agent_name : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘agentName’, alias_priority=2, description=‘Which upstream agent produced this turn. Shown to the model in place of the bare role.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

name : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Filename of the code file.’, examples=[‘clean.py’])]

content : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Code file content aka the code.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

files : list[CleanerAgentCodeFile]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

id : UUID

title : str

turns : list[ConversationTurnResponse]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

conversations : list[ConversationSummaryResponse]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

id : UUID

title : str

created_at : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘createdAt’, alias_priority=2, description=‘Creation time as epoch milliseconds.’)]

updated_at : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘updatedAt’, alias_priority=2, description=‘Last-updated time as epoch milliseconds.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

run_id : Annotated[UUID, FieldInfo(annotation=NoneType, required=True, alias=‘runId’, alias_priority=2)]

question : str

status : Status

result : Annotated[dict[str, Any] | None, FieldInfo(annotation=NoneType, required=True, description=“Rendered AutoRag result for completed turns, in the same shape as POST /query/autorag’s response body.”)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

data_description : Annotated[DataDescription | None, FieldInfo(annotation=NoneType, required=True, alias=‘dataDescription’, alias_priority=2)]

schema_definition : Annotated[SchemaDefinition | None, FieldInfo(annotation=NoneType, required=True, alias=‘schemaDefinition’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

expires_at : str | None

name : str

access_restricted_to : Annotated[list[TokenPermissionEntry] | None, FieldInfo(annotation=NoneType, required=True, alias=‘accessRestrictedTo’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

id : Annotated[UUID, FieldInfo(annotation=NoneType, required=True, description=‘Ingestion id created for background ingestion.’, examples=[‘550e8400-e29b-41d4-a716-446655440000’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

namespace : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The namespace containing the dataset(s) to curate.’, examples=[‘my_namespace’])]

model : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘Optional. The LLM model to use for curation (e.g., “gpt-4”, “claude-3-5-sonnet-20241022”).’, examples=[‘gpt-4’])]

provider : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘Optional. The LLM provider to use (e.g., “openai”, “anthropic”).’, examples=[‘openai’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

datasets_processed : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘datasetsProcessed’, alias_priority=2, description=‘Number of datasets that were processed.’, examples=[5])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

id : str

username : str

name : str | None

email : str | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootCollectionModel[list[dict[str, Any]]]

root : Annotated[list[dict[str, Any]], FieldInfo(annotation=NoneType, required=True, description=‘Array of ontology data objects retrieved from queries.’)]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootModel[str]

root : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘A description of the dataset to help the AI system understand the context.’, examples=[‘Employee dataset containing demographics and compensation.’], metadata=[MaxLen(max_length=10000)])]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

dataset : ApiDatasetDescription

metadata : ApiDatasetMetadata

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootCollectionModel[list[ApiDatasetDescription]]

root : list[ApiDatasetDescription]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

error_code : Annotated[str, FieldInfo(annotation=NoneType, required=True, examples=[‘UNAUTHORIZED’])]

error_message : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, examples=[”])]

message : Annotated[str, FieldInfo(annotation=NoneType, required=True, examples=[‘Missing authentication token’])]

sub_errors : list[SubError] | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

min_distinct : Annotated[float | None, FieldInfo(annotation=NoneType, required=True, alias=‘minDistinct’, alias_priority=2)]

min_variation : Annotated[float | None, FieldInfo(annotation=NoneType, required=True, alias=‘minVariation’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

files_to_upload : Annotated[list[FileUploadPrepareSessionRequestFileToUpload], FieldInfo(annotation=NoneType, required=True, alias=‘filesToUpload’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

filename : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Name of the file to be uploaded.’)]

size : Annotated[int, FieldInfo(annotation=NoneType, required=True, description=‘Size of the file in bytes.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

session_id : Annotated[UploadSessionId, FieldInfo(annotation=NoneType, required=True, alias=‘sessionId’, alias_priority=2)]

files : list[FileUploadPrepareSessionResponseFileInfo]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

file_id : Annotated[UploadFileId, FieldInfo(annotation=NoneType, required=True, alias=‘fileId’, alias_priority=2)]

upload_id : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘uploadId’, alias_priority=2, description=‘Upload identifier’)]

key : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Name of the file to be uploaded.’)]

filename : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘File name’)]

part_size : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘partSize’, alias_priority=2, description=‘Size of the part to be uploaded’)]

presigned_urls : Annotated[list[FileUploadPrepareSessionResponsePresignedInfo], FieldInfo(annotation=NoneType, required=True, alias=‘presignedUrls’, alias_priority=2, description=‘List of presiggned ULR where file has to be uploaded’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

url : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘URL for the presigned upload.’)]

part_size : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘partSize’, alias_priority=2, description=‘Size of the part to be uploaded’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

upload_id : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘uploadId’, alias_priority=2)]

key : str

parts : list[FinishFileUploadRequestParts]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

part_number : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘partNumber’, alias_priority=2)]

etag : str

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

session_id : Annotated[UploadSessionId, FieldInfo(annotation=NoneType, required=True, alias=‘sessionId’, alias_priority=2)]

session_status : Annotated[SessionStatus, FieldInfo(annotation=NoneType, required=True, alias=‘sessionStatus’, alias_priority=2, description=‘Status of the upload session.’, examples=[‘preparing’])]

file_id : Annotated[UploadFileId, FieldInfo(annotation=NoneType, required=True, alias=‘fileId’, alias_priority=2)]

file_status : Annotated[FileStatus, FieldInfo(annotation=NoneType, required=True, alias=‘fileStatus’, alias_priority=2, description=‘Status of the file upload.’, examples=[‘completed’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

part_number : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘partNumber’, alias_priority=2, description=‘Part number (1-based)’, examples=[1])]

etag : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘ETag returned by S3 after uploading the part.’, examples=[‘d41d8cd98f00b204e9800998ecf8427e’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

upload_id : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘uploadId’, alias_priority=2, description=‘Upload ID from the prepare response.’, examples=[‘2~VmxqYXRlRDZWfjJrRWRiZ2FEQXk3dHI4DGhGLygtTmQNXEZPQXFzMVZOZTUgdEdrXCEyNStkQUp3U3Q2RjJqSkV4MFNBOWxJCk0yIHEkfcQ’])]

key : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘S3 object key from the prepare response.’, examples=[‘uploads/data/20240301-143022-uuid.json’])]

parts : Annotated[list[FinishedMultipartItem], FieldInfo(annotation=NoneType, required=True, description=‘Array of completed parts with their ETags.’, metadata=[MinLen(min_length=1)])]

name : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘Optional name/label for the ingestion (e.g., original filename).’, examples=[‘sales-data-2024.csv’])]

infer : Annotated[IngestRequestInfer | None, FieldInfo(annotation=NoneType, required=True, description=‘Optional inference pipeline steps to run on each batch of data after reading from S3.’)]

infer_batch_size : Annotated[int | None, FieldInfo(annotation=NoneType, required=True, alias=‘inferBatchSize’, alias_priority=2, description=‘Number of rows per LLM inference batch. Defaults to 1000. Reduce for large files to avoid S3 read timeouts caused by slow LLM processing.’, examples=[100])]

data_description : Annotated[DataDescription | None, FieldInfo(annotation=NoneType, required=True, alias=‘dataDescription’, alias_priority=2)]

schema_definition : Annotated[SchemaDefinition | None, FieldInfo(annotation=NoneType, required=True, alias=‘schemaDefinition’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

distance : float | None

distance_unit : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘distanceUnit’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

sample : DataArray

model : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The inference model to use.’)]

provider : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘Provider offering the model.’)]

current_value : Annotated[CurrentValue | None, FieldInfo(annotation=NoneType, required=True, alias=‘currentValue’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

description : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘The inferred description text.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

schema_ : Annotated[dict[str, str], FieldInfo(annotation=NoneType, required=True, alias=‘schema’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

indexed : Annotated[int, FieldInfo(annotation=NoneType, required=True, description=‘Number of records indexed by this request.’, examples=[42])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

exact_match : Annotated[ExactMatchConfig | ResolutionParameters | None, FieldInfo(annotation=NoneType, required=True, alias=‘ExactMatch’, alias_priority=2)]

geo_match : Annotated[GeoMatchConfig | ResolutionParameters | None, FieldInfo(annotation=NoneType, required=True, alias=‘GeoMatch’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

data : DataArray

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootCollectionModel[list[ApiLink]]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootCollectionModel[list[dict[str, Any]]]

root : Any]], FieldInfo(annotation=NoneType, required=True, examples=[[{‘id’: ‘001’, ‘__metadata_textToTable’: ‘name -> John Smith, age -> 28, department -> Engineering’, ‘name’: ‘John Smith’, ‘age’: ‘28’, ‘gender’: ‘Male’, ‘department’: ‘Engineering’, ‘location’: ‘New York’}, {‘id’: ‘002’, ‘__metadata_textToTable’: ‘name -> Sarah Johnson, age -> 32, department -> Marketing’, ‘name’: ‘Sarah Johnson’, ‘age’: ‘32’, ‘gender’: ‘Female’, ‘department’: ‘Marketing’, ‘location’: ‘California’}, {‘id’: ‘003’, ‘name’: ‘Michael Brown’, ‘age’: ‘45’, ‘gender’: ‘Male’, ‘department’: ‘Finance’, ‘location’: ‘Texas’, ‘__metadata_normalize’: “Normalized department field from ‘Fin Dept’ to ‘Finance’”}]])]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

data : list[dict[str, Any]]

infer : IngestRequestInfer | None

curate : bool | None

data_description : Annotated[DataDescription | None, FieldInfo(annotation=NoneType, required=True, alias=‘dataDescription’, alias_priority=2)]

schema_definition : Annotated[SchemaDefinition | None, FieldInfo(annotation=NoneType, required=True, alias=‘schemaDefinition’, alias_priority=2)]

additional_instructions : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘additionalInstructions’, alias_priority=2, description=‘Additional instructions to guide the AI system during ingestion.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

steps : list[IngestRequestInferStep] | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootModel[Union[TableStep, RowsStep, NormaliseStep, ValidateStep, ReverseGeoStep, NamedEntitiesStep]]

root : Annotated[TableStep | RowsStep | NormaliseStep | ValidateStep | ReverseGeoStep | NamedEntitiesStep, FieldInfo(annotation=NoneType, required=True, discriminator=‘type’)]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

name : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘Filename or identifier for the ingestion.’)]

seen_at : Annotated[AwareDatetime | None, FieldInfo(annotation=NoneType, required=True, alias=‘seenAt’, alias_priority=2, description=‘Timestamp when the ingestion was first seen.’)]

original_ingestion_id : Annotated[UUID | None, FieldInfo(annotation=NoneType, required=True, alias=‘originalIngestionId’, alias_priority=2, description=“In case of retry, it’s the original ingestion id.”)]

last_retry_at : Annotated[AwareDatetime | None, FieldInfo(annotation=NoneType, required=True, alias=‘lastRetryAt’, alias_priority=2, description=‘Timestamp of the last retry attempt.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

data : list[ListIngestionsResponseDataItem]

metadata : ListIngestionsResponseMetadata

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

id : UUID

status : str

status_message : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘statusMessage’, alias_priority=2)]

processed_bytes : Annotated[int | None, FieldInfo(annotation=NoneType, required=True, alias=‘processedBytes’, alias_priority=2)]

expected_total_bytes : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘expectedTotalBytes’, alias_priority=2)]

processed_rows : Annotated[int | None, FieldInfo(annotation=NoneType, required=True, alias=‘processedRows’, alias_priority=2)]

attributes : IngestionItemAttributes | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

before_created_at : Annotated[AwareDatetime | None, FieldInfo(annotation=NoneType, required=True, alias=‘beforeCreatedAt’, alias_priority=2, description=‘Value of the created_at field before or equal to which return ingestions records.’, metadata=[Ge(ge=1)])]

page_size : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘pageSize’, alias_priority=2, metadata=[Ge(ge=10), Le(le=100)])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

session_id : Annotated[UploadSessionId, FieldInfo(annotation=NoneType, required=True, alias=‘sessionId’, alias_priority=2)]

session_status : Annotated[SessionStatus, FieldInfo(annotation=NoneType, required=True, alias=‘sessionStatus’, alias_priority=2, description=‘Status of the upload session.’, examples=[‘completed’])]

files : list[ListSessionFilesResponseFileInfo]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

file_id : Annotated[UploadFileId, FieldInfo(annotation=NoneType, required=True, alias=‘fileId’, alias_priority=2)]

filename : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Original filename provided during prepare.’)]

s3_bucket : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘s3Bucket’, alias_priority=2, description=‘S3 bucket containing the uploaded object.’)]

s3_key : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘s3Key’, alias_priority=2, description=‘S3 key of the uploaded object.’)]

file_size : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘fileSize’, alias_priority=2, description=‘Size of the uploaded file in bytes.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

type : Literal[‘namedEntities’]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

type : Literal[‘normalise’]

mode : Annotated[Mode, FieldInfo(annotation=NoneType, required=True, description=‘Normalisation mode.’)]

target_cols : Annotated[dict[str, str] | None, FieldInfo(annotation=NoneType, required=True, alias=‘targetCols’, alias_priority=2, description=‘Target columns with their descriptions.’)]

data_description : Annotated[DataDescription | None, FieldInfo(annotation=NoneType, required=True, alias=‘dataDescription’, alias_priority=2)]

model : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The LLM model to use for this step.’)]

provider : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The LLM provider to use for this step.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootCollectionModel[list[ApiLink]]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootModel[str]

root : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Type for the result of an operation.’)]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

filename : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Name of the file being uploaded.’, examples=[‘large-data.csv’])]

size : Annotated[int, FieldInfo(annotation=NoneType, required=True, description=‘Size of the file in bytes.’, examples=[73473457])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

upload_id : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘uploadId’, alias_priority=2, description=‘Unique identifier for the multipart upload session.’, examples=[‘2~VmxqYXRlRDZWfjJrRWRiZ2FEQXk3dHI4DGhGLygtTmQNXEZPQXFzMVZOZTUgdEdrXCEyNStkQUp3U3Q2RjJqSkV4MFNBOWxJCk0yIHEkfcQ’])]

key : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Generated S3 object key.’, examples=[‘uploads/data/20240301-143022-uuid.json’])]

part_size : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘partSize’, alias_priority=2, description=‘Size of each part in bytes (8MB).’, examples=[8388608])]

presigned_urls : Annotated[list[PresignedUrlItem], FieldInfo(annotation=NoneType, required=True, alias=‘presignedUrls’, alias_priority=2, description=‘Array of presigned URLs for multipart upload.’, metadata=[MinLen(min_length=1), MaxLen(max_length=10000)])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

url : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Presigned URL for uploading this part.’, examples=[‘https://my-data-bucket.s3.amazonaws.com/uploads/data/20240301-143022-uuid.json?partNumber=1&uploadId=…’])]

size : Annotated[int, FieldInfo(annotation=NoneType, required=True, description=‘Exact number of bytes to upload to this URL. Use this value to slice the file — the last part will be smaller than partSize when the file size is not a multiple of partSize.’, examples=[8388608])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

username : str

namespace : str

dataset : str | None

query : str

natural_language_query : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘naturalLanguageQuery’, alias_priority=2)]

model : str | None

provider : str | None

explain_requested : Annotated[bool | None, FieldInfo(annotation=NoneType, required=True, alias=‘explainRequested’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

id : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Id of the step.’, examples=[‘930f9d0d-eb0e-4e2f-9b1f-91855efe6b94’])]

dataset_queries_stat : Annotated[QueryExplainResponseDatasetQueriesStat, FieldInfo(annotation=NoneType, required=True, alias=‘datasetQueriesStat’, alias_priority=2)]

start : Annotated[AwareDatetime, FieldInfo(annotation=NoneType, required=True, description=‘Timestamp of the step start.’, examples=[‘2024-03-01T14:30:22.000Z’])]

duration : Annotated[timedelta, FieldInfo(annotation=NoneType, required=True, description=‘Duration of the step in ISO 8601 format.’, examples=[‘PT1M30S’])]

children : list[QueryExplainResponse]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

username : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Username that dataset belong to’)]

namespace : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Namespace dataset belongs to.’)]

dataset : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Name of dataset.’)]

result_count : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘resultCount’, alias_priority=2, description=‘Number of results returned from the dataset.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

data : DataArray

explain_results : Annotated[QueryExplainResponse | None, FieldInfo(annotation=NoneType, required=True, alias=‘explainResults’, alias_priority=2)]

metadata : QueryResponseMetadata

result_type : Annotated[OperationResult | None, FieldInfo(annotation=NoneType, required=True, alias=‘resultType’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

count : Annotated[int | None, FieldInfo(annotation=NoneType, required=True, description=‘Number of results returned.’)]

executed_query : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘executedQuery’, alias_priority=2, description=‘The actual query that was executed.’)]

original_query : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘originalQuery’, alias_priority=2, description=‘The original query provided by the user.’)]

starting_dataset : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘startingDataset’, alias_priority=2, description=‘The initial dataset used when evaluating the query. For linked queries, this is the dataset where the query begins.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

prompts : Annotated[list[str], FieldInfo(annotation=NoneType, required=True, description=‘Prompts that describe each cleaning step in the intended order.’, examples=[[‘Remove extra spaces from firstName and lastName fields.’, ‘Remove empty values from the age field.’]], metadata=[MinLen(min_length=1), MaxLen(max_length=10)])]

output_namespace : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘outputNamespace’, alias_priority=2, description=‘Target namespace where the cleaned dataset will be created.’, examples=[‘all_about_movies’])]

output_dataset_name : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘outputDatasetName’, alias_priority=2, description=‘Specifies the name of the cleaned dataset. The service creates this dataset when the cleaning job starts, and the name must be unused in the target namespace.’, examples=[‘movies_cleaned’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

cleaning_task_id : Annotated[UUID, FieldInfo(annotation=NoneType, required=True, alias=‘cleaningTaskId’, alias_priority=2, description=‘Id for cleaning task.’, examples=[‘550e8400-e29b-41d4-a716-446655440000’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

bidirectional : bool | None

target_columns : Annotated[list[str] | None, FieldInfo(annotation=NoneType, required=True, alias=‘targetColumns’, alias_priority=2)]

ignore_private_columns : Annotated[bool | None, FieldInfo(annotation=NoneType, required=True, alias=‘ignorePrivateColumns’, alias_priority=2)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

type : Literal[‘reverseGeo’]

derive_from : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘deriveFrom’, alias_priority=2, description=‘Name of the location column.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

type : Literal[‘rows’]

derive_from : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘deriveFrom’, alias_priority=2, description=‘Name of the column that contains the JSON content.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootCollectionModel[dict[str, str]]

root : dict[str, str]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

username : Annotated[str, FieldInfo(annotation=NoneType, required=True, description=‘Username of the user to grant or revoke access for.’, examples=[‘alice’])]

role : Annotated[Role, FieldInfo(annotation=NoneType, required=True, description=“Access level. ‘viewer’ is read-only; ‘editor’ is read and write.”, examples=[‘viewer’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

type : Literal[‘table’]

derive_from : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘deriveFrom’, alias_priority=2, description=‘Name of the source column to extract from.’)]

additional_instructions : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, alias=‘additionalInstructions’, alias_priority=2, description=‘Additional instructions to guide the AI system for higher accuracy during extraction.’, examples=[“This text contains a table comma separated, but instead of line breaks we are using a ’;’.”])]

model : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The LLM model to use for this step.’)]

provider : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The LLM provider to use for this step.’)]

replace_original_column : Annotated[bool, FieldInfo(annotation=NoneType, required=True, alias=‘replaceOriginalColumn’, alias_priority=2, description=‘When true, the source column specified in deriveFrom is removed from the output after extraction.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

username : str

namespace : str

dataset : str | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

restricted : bool

permissions : list[TokenPermissionEntry] | None

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

order : Annotated[list[str], FieldInfo(annotation=NoneType, required=True, description=‘Ordered list of all column names in the desired sequence.’, examples=[[‘id’, ‘name’, ‘age’, ‘department’, ‘location’]])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootModel[UUID]

root : Annotated[UUID, FieldInfo(annotation=NoneType, required=True, description=‘Unique identifier for a file’, examples=[‘311fe2cb-1c85-46a4-a8d9-f63a88da44fd’])]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootModel[UUID]

root : Annotated[UUID, FieldInfo(annotation=NoneType, required=True, description=‘Unique identifier for the upload session.’, examples=[‘24b49ba0-6b5c-4950-94c1-1622939cc481’])]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

created_at : str

expires_at : str

id : str

last_used_at : str | None

name : str

token : Annotated[str | None, FieldInfo(annotation=NoneType, required=True)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RootCollectionModel[list[UserToken]]

root : list[UserToken]

model_config : ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

date : Annotated[date, FieldInfo(annotation=NoneType, required=True, description=‘The day of the aggregated usage.’, examples=[‘2021-01-21’])]

tokens_used : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘tokensUsed’, alias_priority=2, description=‘Total number of tokens used in this day.’, examples=[125000])]

cost_usd : Annotated[float, FieldInfo(annotation=NoneType, required=True, alias=‘costUSD’, alias_priority=2, description=‘Total cost in USD for this day.’, examples=[12.5])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

time_zone_for_aggregation : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘timeZoneForAggregation’, alias_priority=2, description=‘The timezone used for date aggregation.’, examples=[‘America/New_York’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

data : list[UserUsageByDayItem]

meta : UserUsageByDayMeta

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

last_start_time : Annotated[AwareDatetime, FieldInfo(annotation=NoneType, required=True, alias=‘lastStartTime’, alias_priority=2, description=‘The start time of the last observation in the current page.’, examples=[‘2021-01-01T12:12:12.000Z’])]

last_observation_id : Annotated[str, FieldInfo(annotation=NoneType, required=True, alias=‘lastObservationId’, alias_priority=2, description=‘The observation ID of the last observation in the current page.’, examples=[‘obs_12345’])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

timestamp : Annotated[AwareDatetime, FieldInfo(annotation=NoneType, required=True, description=‘The timestamp of the usage record.’, examples=[‘2021-01-01T12:12:12.000Z’])]

tokens_used : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘tokensUsed’, alias_priority=2, description=‘Number of tokens used.’, examples=[1241])]

cost_usd : Annotated[float, FieldInfo(annotation=NoneType, required=True, alias=‘costUSD’, alias_priority=2, description=‘Cost in USD for this usage.’, examples=[0.0001])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

page_cursor : Annotated[UserUsageHistoryCursor | None, FieldInfo(annotation=NoneType, required=True, alias=‘pageCursor’, alias_priority=2, description=‘Cursor for fetching the next page of results.’)]

page_size : Annotated[int, FieldInfo(annotation=NoneType, required=True, alias=‘pageSize’, alias_priority=2, description=‘Number of items per page.’, examples=[100])]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: ResponseModel

data : list[UserUsageHistoryItem]

meta : UserUsageHistoryMeta

model_config : ClassVar[ConfigDict] = {‘extra’: ‘allow’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: RequestModel

type : Literal[‘validate’]

mode : Annotated[Mode1, FieldInfo(annotation=NoneType, required=True, description=‘Validation mode.’)]

columns : Annotated[list[str], FieldInfo(annotation=NoneType, required=True, description=‘Columns to validate.’)]

model : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The LLM model to use for this step.’)]

provider : Annotated[str | None, FieldInfo(annotation=NoneType, required=True, description=‘The LLM provider to use for this step.’)]

model_config : ClassVar[ConfigDict] = {‘extra’: ‘forbid’, ‘populate_by_name’: True, ‘validate_by_alias’: True, ‘validate_by_name’: True}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict]. Bases: Enum Represents different types of processing steps for data manipulation. This class enumerates various distinct processing types that can be used in DataLinks workflows. Each enumeration value signifies a specific stage in the broader data-processing pipeline.

TABLE = ‘table’

ROWS = ‘rows’

NORMALIZE = ‘normalise’

VALIDATE = ‘validate’

REVERSE_GEO = ‘reverseGeo’

Bases: Enum Enumeration for normalization modes. This class represents different modes of data normalization used in the ‘normalize’ step. It provides three options for normalization: ‘embeddings’ for embedding-level normalization, ‘all-in-one’ for holistic normalization, and ‘field-by-field’ for column-wise normalization.
  • Variables:
    • EMBEDDINGS – Mode for normalizing data on an embedding level.
    • ALL_IN_ONE – Mode for normalizing data holistically, treating the entire dataset as a single entity.
    • FIELD_BY_FIELD – Mode for normalizing data column-by-column, focusing on individual fields independently.

EMBEDDINGS = ‘embeddings’

ALL_IN_ONE = ‘all-in-one’

FIELD_BY_FIELD = ‘field-by-field’

Bases: Enum Enumeration class that defines various validation modes. This class is designed to specify the modes of operation for the ‘validate’ step. The predefined modes include validation by rows, regular expressions, and fields.
  • Variables:
    • ROWS – Validation mode that focuses on rows.
    • REGEX – Validation mode that utilizes regular expressions.
    • FIELDS – Validation mode that focuses on columns.

ROWS = ‘rows’

REGEX = ‘regex’

FIELDS = ‘fields’

Bases: object Represents the base step within DataLinks. This class serves as the foundational step structure for various implementations. It includes methods to transform its data representation into a dictionary format, custom-processed with specific rules for attributes of Enum type. It is primarily designed as a metaclass.
  • Variables: step_type – The type of the step, categorized using StepTypes.

step_type : ClassVar[StepTypes]

to_dict()

  • Return type: dict
Bases: BaseStep Common class for pipeline steps that rely on LLM inference.
  • Variables:
    • model – The name of the model to use in the step.
    • provider – The identifier of the provider to be used (ollama, openai, etc)

model : str | None

provider : str | None

Bases: BaseStep Represents the ‘infer’ step in the DataLinks workflow.
  • Variables: derive_from – The identifier of the source field used in the inference step.

derive_from : str

Bases: LlmStep, InferenceStep Use this step to infer a table from unstructured data.
  • Variables: helper_prompt – A string that stores an optional helper prompt or additional guiding context specific to the table inference step.

step_type : ClassVar[StepTypes] = ‘table’

helper_prompt : str = ”

Bases: InferenceStep Use this step to extract data that is already in tabular format (eg.: CSV).

step_type : ClassVar[StepTypes] = ‘rows’

Bases: InferenceStep Use this step to perform reverse geolocation based on the source field.

step_type : ClassVar[StepTypes] = ‘reverseGeo’

Bases: LlmStep Use this step to attempt normalisation of the extracted column names. Table inference across different unstructured data blocks may result in different field names for the same information, hence the need to normalize the column names. Encapsulates the configuration necessary to perform the ‘normalize’ step. It specifies the desired target columns, the mode of normalisation, and includes optional helper prompts to provide further instructions or context.
  • Variables:
    • target_cols – A mapping of the desired column names to an optional description used as context.
    • mode – Specifies the normalisation mode to be applied.
    • helper_prompt – Optional helper text or prompt information.

step_type : ClassVar[StepTypes] = ‘normalise’

target_cols : Mapping[str, str | None]

mode : NormalizeModes

helper_prompt : str = ”

Bases: LlmStep Use this step to add data validation to the inference pipeline.
  • Variables:
    • mode – Indicates the mode of validation to be applied.
    • columns – List containing the column names which are used for validation.

step_type : ClassVar[StepTypes] = ‘validate’

mode : ValidateModes

columns : List[str]

Bases: object Represents a collection of sequential steps. Holds and manages the sequence of steps used for ingesting and/or enhancing data.
  • Variables: steps – A collection of steps to be executed in sequence.

to_list()

  • Return type: list[dict[str, Any]]