Set the dataframe info which needs to be visualized.

Description

Set the dataframe info which needs to be visualized.

Usage

display(dataFrame, isSummary = FALSE)

Arguments

Value

No return value, show the first part of passed dataFrame.

Examples

data <- list(56,78,90,45,67)
df <- data.frame(t(sapply(data,c)))
display(df)
display(df, TRUE)

Set the chart config metadata for current dataframe (set by display) which needs to be visualized.

Description

Set the chart config metadata for current dataframe (set by display) which needs to be visualized.

Usage

display.config(
  commId,
  lastCommId = NULL,
  binsNumber = 10,
  category = "table",
  keys = NULL,
  values = NULL,
  series = NULL,
  aggregation = NULL,
  column = NULL
)

Arguments

Construct an specific html fragment to synapse notebook front-end for rendering based on user-input html content.

Description

Construct an specific html fragment to synapse notebook front-end for rendering based on user-input html content.

Usage

displayHTML(content)

Arguments

Value

No return value, print the content to mimic the render behavior when used in azure synapse runtime.

Examples

displayHTML('<b>Hello world!</b>')

Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.

Description

Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.

Usage

mssparkutils.credentials.getConnectionStringOrCreds(linkedService)

Arguments

Value

A empty string used to mimic credentials returned by azure synapse runtime for linkedService.

Examples

mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')

Take linked service name as input and return full connection string with credentials.

Description

Take linked service name as input and return full connection string with credentials.

Usage

mssparkutils.credentials.getFullConnectionString(linkedService)

Arguments

Value

A empty string used to mimic connection string returned by azure synapse runtime for linkedService.

Examples

mssparkutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')

Return all the properties of a given linked service in string format.

Description

Return all the properties of a given linked service in string format.

Usage

mssparkutils.credentials.getPropertiesAll(linkedService)

Arguments

Value

A empty string used to mimic properties string returned by azure synapse runtime for linkedService.

Examples

mssparkutils.credentials.getPropertiesAll('AzureDataLakeStorage1')

Return AKV secret.

Description

Return AKV secret.

Usage

mssparkutils.credentials.getSecret(akvName, secret, linkedService = NULL)

Arguments

Value

A empty string used to mimic secret returned by azure synapse runtime for given akvName and secret.

Examples

mssparkutils.credentials.getSecret('akvName', 'secretName')
mssparkutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')

Return AKV secret using linkedService.

Description

Return AKV secret using linkedService.

Usage

mssparkutils.credentials.getSecretWithLS(linkedService, secret)

Arguments

Value

A empty string used to mimic secret returned by azure synapse runtime for given linkedService and secret.

Examples

mssparkutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')

Get AAD token for a resource.

Description

Get AAD token for a resource.

Usage

mssparkutils.credentials.getToken(audience, name = "")

Arguments

Value

A empty string used to mimic token returned by azure synapse runtime for accessing resource audience.

Examples

mssparkutils.credentials.getToken('synapse')
mssparkutils.credentials.getToken('storage')
mssparkutils.credentials.getToken('storage', 'storage')

Get help message.

Description

Get help message.

Usage

mssparkutils.credentials.help()

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils credentials module when used in azure synapse runtime.

Examples

mssparkutils.credentials.help()

Returns true if the input token is valid (i.e, hasn't expired).

Description

Returns true if the input token is valid (i.e, hasn't expired).

Usage

mssparkutils.credentials.isValidToken(token)

Arguments

Value

FALSE to mimic the result if token is invalid.

Examples

mssparkutils.credentials.isValidToken('dummyToken')

Put AKV secret using with or without linkedService.

Description

Put AKV secret using with or without linkedService.

Usage

mssparkutils.credentials.putSecret(
  akvName,
  secretName,
  secretValue,
  linkedService = NULL
)

Arguments

Value

The secretValue been written.

Examples

mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue')
mssparkutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')

Put AKV secret using linkedService.

Description

Put AKV secret using linkedService.

Usage

mssparkutils.credentials.putSecretWithLS(
  linkedService,
  secretName,
  secretValue
)

Arguments

Value

The secretValue been written.

Examples

mssparkutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')

Get cluster id.

Description

Get cluster id.

Usage

mssparkutils.env.getClusterId()

Value

A empty string used to mimic cluster id of azure synapse runtime.

Examples

mssparkutils.env.getClusterId()

Get job Id.

Description

Get job Id.

Usage

mssparkutils.env.getJobId()

Value

A empty string used to mimic the id of spark job been submitted to azure synapse runtime.

Examples

mssparkutils.env.getJobId()

Get pool name.

Description

Get pool name.

Usage

mssparkutils.env.getPoolName()

Value

A empty string used to mimic the name of user's azure synapse spark pool.

Examples

mssparkutils.env.getPoolName()

Get user Id.

Description

Get user Id.

Usage

mssparkutils.env.getUserId()

Value

A empty string used to mimic the id of user.

Examples

mssparkutils.env.getUserId()

Get user name.

Description

Get user name.

Usage

mssparkutils.env.getUserName()

Value

A empty string used to mimic the name of user.

Examples

mssparkutils.env.getUserName()

Get workspace name.

Description

Get workspace name.

Usage

mssparkutils.env.getWorkspaceName()

Value

A empty string used to mimic the id of the user's azure synapse workspace.

Examples

mssparkutils.env.getWorkspaceName()

Get help message.

Description

Get help message.

Usage

mssparkutils.env.help()

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils env module when used in azure synapse runtime.

Examples

mssparkutils.env.help()

Append the given String to a file, encoded in UTF-8.

Description

Append the given String to a file, encoded in UTF-8.

Usage

mssparkutils.fs.append(file, content, createFileIfNotExists = FALSE)

Arguments

Value

FALSE to mimic the result if file content append fail.

Examples

mssparkutils.fs.append("/tmp/my-file", "Hello world!")
mssparkutils.fs.append("/tmp/my-file", "Hello world!", TRUE)

Copies a file or directory, possibly across FileSystems.

Description

Copies a file or directory, possibly across FileSystems.

Usage

mssparkutils.fs.cp(from, to, recurse = FALSE)

Arguments

Value

FALSE to mimic the result if file or directory from fail to copy to to.

Examples

mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b")
mssparkutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)

Check if a file or directory exists.

Description

Check if a file or directory exists.

Usage

mssparkutils.fs.exists(file)

Arguments

Value

TRUE if the file or directory exists

Examples

## Not run: 
mssparkutils.fs.exists("/tmp/my-file")

## End(Not run)

Copies a file or directory via azcopy, possibly across FileSystems.

Description

Copies a file or directory via azcopy, possibly across FileSystems.

Usage

mssparkutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)

Arguments

Value

TRUE if all files were successfully copied

Examples

## Not run: 
mssparkutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b")

## End(Not run)

Gets the local path of the mount point.

Description

Gets the local path of the mount point.

Usage

mssparkutils.fs.getMountPath(mountPoint, scope = "job")

Arguments

Value

Empty string to mimic the local mounted path related to mountPoint.

Examples

mssparkutils.fs.getMountPath("/mnt")
mssparkutils.fs.getMountPath("/mnt", "job")

Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.

Description

Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.

Usage

mssparkutils.fs.head(file, maxBytes = 65535)

Arguments

Value

Empty string to mimic the returned content of file.

Examples

mssparkutils.fs.head("/tmp/my-folder/my-file")
mssparkutils.fs.head("/tmp/my-folder/my-file", 1000)

mssparkutils.fs provides utilities for working with various FileSystems.

Description

Below is overview about the available methods:

Usage

mssparkutils.fs.help(methodName = "")

Arguments

Details

mssparkutils.fs.cp: Copies a file or directory, possibly across FileSystems mssparkutils.fs.mv: Moves a file or directory, possibly across FileSystems mssparkutils.fs.ls: Array -> Lists the contents of a directory mssparkutils.fs.mkdirs: Creates the given directory if it does not exist, also creating any necessary parent directories mssparkutils.fs.put: Writes the given String out to a file, encoded in UTF-8 mssparkutils.fs.head: Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8 mssparkutils.fs.append: Append the content to a file mssparkutils.fs.rm: Removes a file or directory

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils fs module when used in azure synapse runtime.

Examples

mssparkutils.fs.help()
mssparkutils.fs.help("ls")

Lists the contents of a directory.

Description

Lists the contents of a directory.

Usage

mssparkutils.fs.ls(dir)

Arguments

Value

Empty list to mimic the file list under dir.

Examples

mssparkutils.fs.ls("/tmp/my-folder/")

Creates the given directory if it does not exist, also creating any necessary parent * directories.

Description

Creates the given directory if it does not exist, also creating any necessary parent * directories.

Usage

mssparkutils.fs.mkdirs(dir)

Arguments

Value

FALSE to mimic the result if dir creation fail.

Examples

mssparkutils.fs.mkdirs("/tmp/a/b/c")

Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)

Description

Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)

Usage

mssparkutils.fs.mount(source, mountPoint, extraConfigs = NULL)

Arguments

Value

FALSE to mimic the result if mountPoint creation fail.

Examples

mssparkutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")

Attach remote storage (Blob, Gen2, Azure File Share) to driver node

Description

Attach remote storage (Blob, Gen2, Azure File Share) to driver node

Usage

mssparkutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)

Arguments

Value

TRUE if the path was successfully mounted.

Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.

Description

Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.

Usage

mssparkutils.fs.mounts(extraConfigs = NULL)

Arguments

Value

The list of MountPointInfo.

Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.

Description

Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.

Usage

mssparkutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)

Arguments

Value

FALSE to mimic the result of mv operation fail.

Examples

mssparkutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")

Writes the given String out to a file, encoded in UTF-8.

Description

Writes the given String out to a file, encoded in UTF-8.

Usage

mssparkutils.fs.put(file, content, overwrite = FALSE)

Arguments

Value

FALSE to mimic the result of file put operation fail.

Examples

mssparkutils.fs.put("/tmp/my-file", "Hello world!", TRUE)

Refresh workspace level mount points.

Description

Refresh workspace level mount points.

Usage

mssparkutils.fs.refreshMounts()

Value

FALSE to mimic the refreshMounts fail to refresh mount info.

Examples

mssparkutils.fs.refreshMounts()

Removes a file or directory.

Description

Removes a file or directory.

Usage

mssparkutils.fs.rm(dir, recurse = FALSE)

Arguments

Value

FALSE to mimic the result of dir deletion fail.

Examples

mssparkutils.fs.rm("/tmp/my-folder/", TRUE)

Removes a mount point.

Description

Removes a mount point.

Usage

mssparkutils.fs.unmount(mountPoint)

Arguments

Value

FALSE to mimic the result of unmount mountPoint fail.

Examples

mssparkutils.fs.unmount("/mnt")

Removes a mount point from driver node.

Description

Removes a mount point from driver node.

Usage

mssparkutils.fs.unmountFromDriverNode(mountPoint)

Arguments

Value

TRUE if the mount point was successfully unmounted.

Get help message for this module.

Description

Get help message for this module.

Usage

mssparkutils.help(methodName = "")

Arguments

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils module when used in azure synapse runtime.

Examples

mssparkutils.help()

Create a lakehouse

Description

Create a lakehouse

Usage

mssparkutils.lakehouse.create(
  name,
  description = "",
  definition = "",
  workspaceId = ""
)

Arguments

Value

A lakehouse object

Delete a lakehouse

Description

Delete a lakehouse

Usage

mssparkutils.lakehouse.delete(name, workspaceId = "")

Arguments

Get a lakehouse

Description

Get a lakehouse

Usage

mssparkutils.lakehouse.get(name = "", workspaceId = "")

Arguments

Value

A lakehouse object

The lakehouse module.

Description

mssparkutils.lakehouse.create(name: String, description: String, workspaceId: String): Lakehouse -> Create a lakehouse mssparkutils.lakehouse.get(name: String, workspaceId: String): Lakehouse -> Get a lakehouse mssparkutils.lakehouse.delete(name: String, workspaceId: String): void -> Delete a lakehouse mssparkutils.lakehouse.update(name: String, newName: String, description: String, workspaceId: String): Lakehouse -> Update a lakehouse

Usage

mssparkutils.lakehouse.help(methodName = "")

Arguments

List all lakehouses

Description

List all lakehouses

Usage

mssparkutils.lakehouse.list(workspaceId = "", maxResults = 1000L)

Arguments

Value

A list of lakehouse objects

Update a lakehouse

Description

Update a lakehouse

Usage

mssparkutils.lakehouse.update(
  name,
  newName,
  description = "",
  workspaceId = ""
)

Arguments

Value

A lakehouse object

This method lets you exit a notebook with a value.

Description

This method lets you exit a notebook with a value.

Usage

mssparkutils.notebook.exit(value)

Arguments

Value

No return value, mimic behavior to set the notebook run exit value using value.

Examples

mssparkutils.notebook.exit('exitVal')

The notebook module.

Description

The notebook module.

Usage

mssparkutils.notebook.help(methodName = "")

Arguments

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils notebook module when used in azure synapse runtime.

Examples

mssparkutils.notebook.help()
mssparkutils.notebook.help("run")

Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.

Description

Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.

Usage

mssparkutils.notebook.run(path, timeoutSeconds = 90, arguments = NULL)

Arguments

Value

Empty string to mimic the exitVal set by mssparkutils.notebook.exit.

Examples

mssparkutils.notebook.run('NB1')
mssparkutils.notebook.run('NB1', 200)
mssparkutils.notebook.run('NB1', 200, list("input"=30))

Runs multiple notebooks concurrently with support for dependency relationships. Details can be found in mssparkutils.notebook.help("runMultiple").

Description

Runs multiple notebooks concurrently with support for dependency relationships. Details can be found in mssparkutils.notebook.help("runMultiple").

Usage

mssparkutils.notebook.runMultiple(pathsOrPipeline)

Arguments

Value

a list of exit values and exceptions for each notebook

provide a way to make people can update the endpoint

Description

provide a way to make people can update the endpoint

Usage

mssparkutils.notebook.updateNBSEndpoint(endpoint)

Arguments

Get runtime properties

Description

Get runtime properties

Usage

mssparkutils.runtime.context()

Value

A dummy env object to mimic the result of runtime context method when used in azure synapse runtime.

Examples

mssparkutils.runtime.context()

Set runtime high concurrency mode repl id

Description

Set runtime high concurrency mode repl id

Usage

mssparkutils.runtime.setHcReplId(replId)

Arguments

Stop an interactive session

Description

Stop an interactive session

Usage

mssparkutils.session.stop(detach = TRUE)

Arguments

Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.

Description

Take linked service name as input and return connection string or credentials depending on the configuration of the linked service.

Usage

notebookutils.credentials.getConnectionStringOrCreds(linkedService)

Arguments

Value

A empty string used to mimic credentials returned by azure synapse runtime for linkedService.

Examples

notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')

Take linked service name as input and return full connection string with credentials.

Description

Take linked service name as input and return full connection string with credentials.

Usage

notebookutils.credentials.getFullConnectionString(linkedService)

Arguments

Value

A empty string used to mimic connection string returned by azure synapse runtime for linkedService.

Examples

notebookutils.credentials.getConnectionStringOrCreds('AzureDataLakeStorage1')

Return all the properties of a given linked service in string format.

Description

Return all the properties of a given linked service in string format.

Usage

notebookutils.credentials.getPropertiesAll(linkedService)

Arguments

Value

A empty string used to mimic properties string returned by azure synapse runtime for linkedService.

Examples

notebookutils.credentials.getPropertiesAll('AzureDataLakeStorage1')

Return AKV secret.

Description

Return AKV secret.

Usage

notebookutils.credentials.getSecret(akvName, secret, linkedService = NULL)

Arguments

Value

A empty string used to mimic secret returned by azure synapse runtime for given akvName and secret.

Examples

notebookutils.credentials.getSecret('akvName', 'secretName')
notebookutils.credentials.getSecret('akvName', 'secretName', 'AzureDataLakeStorage1')

Return AKV secret using linkedService.

Description

Return AKV secret using linkedService.

Usage

notebookutils.credentials.getSecretWithLS(linkedService, secret)

Arguments

Value

A empty string used to mimic secret returned by azure synapse runtime for given linkedService and secret.

Examples

notebookutils.credentials.getSecretWithLS('AzureDataLakeStorage1', 'secretName')

Get AAD token for a resource.

Description

Get AAD token for a resource.

Usage

notebookutils.credentials.getToken(audience, name = "")

Arguments

Value

A empty string used to mimic token returned by azure synapse runtime for accessing resource audience.

Examples

notebookutils.credentials.getToken('synapse')
notebookutils.credentials.getToken('storage')
notebookutils.credentials.getToken('storage', 'storage')

Get help message.

Description

Get help message.

Usage

notebookutils.credentials.help()

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils credentials module when used in azure synapse runtime.

Examples

notebookutils.credentials.help()

Returns true if the input token is valid (i.e, hasn't expired).

Description

Returns true if the input token is valid (i.e, hasn't expired).

Usage

notebookutils.credentials.isValidToken(token)

Arguments

Value

FALSE to mimic the result if token is invalid.

Examples

notebookutils.credentials.isValidToken('dummyToken')

Put AKV secret using with or without linkedService.

Description

Put AKV secret using with or without linkedService.

Usage

notebookutils.credentials.putSecret(
  akvName,
  secretName,
  secretValue,
  linkedService = NULL
)

Arguments

Value

The secretValue been written.

Examples

notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue')
notebookutils.credentials.putSecret('akvName', 'secretName', 'secretValue', 'AzureDataLakeStorage1')

Put AKV secret using linkedService.

Description

Put AKV secret using linkedService.

Usage

notebookutils.credentials.putSecretWithLS(
  linkedService,
  secretName,
  secretValue
)

Arguments

Value

The secretValue been written.

Examples

notebookutils.credentials.putSecretWithLS('AzureDataLakeStorage1', 'secretName', 'secretValue')

Get cluster id.

Description

Get cluster id.

Usage

notebookutils.env.getClusterId()

Value

A empty string used to mimic cluster id of azure synapse runtime.

Examples

notebookutils.env.getClusterId()

Get job Id.

Description

Get job Id.

Usage

notebookutils.env.getJobId()

Value

A empty string used to mimic the id of spark job been submitted to azure synapse runtime.

Examples

notebookutils.env.getJobId()

Get pool name.

Description

Get pool name.

Usage

notebookutils.env.getPoolName()

Value

A empty string used to mimic the name of user's azure synapse spark pool.

Examples

notebookutils.env.getPoolName()

Get user Id.

Description

Get user Id.

Usage

notebookutils.env.getUserId()

Value

A empty string used to mimic the id of user.

Examples

notebookutils.env.getUserId()

Get user name.

Description

Get user name.

Usage

notebookutils.env.getUserName()

Value

A empty string used to mimic the name of user.

Examples

notebookutils.env.getUserName()

Get workspace name.

Description

Get workspace name.

Usage

notebookutils.env.getWorkspaceName()

Value

A empty string used to mimic the id of the user's azure synapse workspace.

Examples

notebookutils.env.getWorkspaceName()

Get help message.

Description

Get help message.

Usage

notebookutils.env.help()

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils env module when used in azure synapse runtime.

Examples

notebookutils.env.help()

Send a DELETE request to Fabric.

Description

Send a DELETE request to Fabric.

Usage

notebookutils.fabricClient.delete(path, headers = list())

Arguments

Send a GET request to Fabric.

Description

Send a GET request to Fabric.

Usage

notebookutils.fabricClient.get(path, headers = list())

Arguments

Value

RestResponse Response of the request

Get help string for a method.

Description

Get help string for a method.

Usage

notebookutils.fabricClient.help(methodName = "")

Arguments

List all capacities in the workspace.

Description

List all capacities in the workspace.

Usage

notebookutils.fabricClient.listCapacities(maxResults = 1000L)

Arguments

Value

Array of Capacity objects

Send a PATCH request to Fabric.

Description

Send a PATCH request to Fabric.

Usage

notebookutils.fabricClient.patch(path, content, headers = list())

Arguments

Value

RestResponse Response of the request

Send a POST request to Fabric.

Description

Send a POST request to Fabric.

Usage

notebookutils.fabricClient.post(path, content, headers = list())

Arguments

Value

RestResponse Response of the request

Send a PUT request to Fabric.

Description

Send a PUT request to Fabric.

Usage

notebookutils.fabricClient.put(path, content, headers = list())

Arguments

Value

RestResponse Response of the request

Append the given String to a file, encoded in UTF-8.

Description

Append the given String to a file, encoded in UTF-8.

Usage

notebookutils.fs.append(file, content, createFileIfNotExists = FALSE)

Arguments

Value

FALSE to mimic the result if file content append fail.

Examples

notebookutils.fs.append("/tmp/my-file", "Hello world!")
notebookutils.fs.append("/tmp/my-file", "Hello world!", TRUE)

Copies a file or directory, possibly across FileSystems.

Description

Copies a file or directory, possibly across FileSystems.

Usage

notebookutils.fs.cp(from, to, recurse = FALSE)

Arguments

Value

FALSE to mimic the result if file or directory from fail to copy to to.

Examples

notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b")
notebookutils.fs.cp("/tmp/my-folder/a", "adls://xxx/tmp/b", TRUE)

Check if a file or directory exists.

Description

Check if a file or directory exists.

Usage

notebookutils.fs.exists(file)

Arguments

Value

TRUE if the file or directory exists

Examples

## Not run: 
notebookutils.fs.exists("/tmp/my-file")

## End(Not run)

Copies a file or directory via azcopy, possibly across FileSystems.

Description

Copies a file or directory via azcopy, possibly across FileSystems.

Usage

notebookutils.fs.fastcp(from, to, recurse = TRUE, extraConfigs = NULL)

Arguments

Value

TRUE if all files were successfully copied

Examples

## Not run: 
notebookutils.fs.fastcp("file:/tmp/my-folder/a", "adls://xxx/tmp/b")

## End(Not run)

Gets the local path of the mount point.

Description

Gets the local path of the mount point.

Usage

notebookutils.fs.getMountPath(mountPoint, scope = "job")

Arguments

Value

Empty string to mimic the local mounted path related to mountPoint.

Examples

notebookutils.fs.getMountPath("/mnt")
notebookutils.fs.getMountPath("/mnt", "job")

Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.

Description

Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8.

Usage

notebookutils.fs.head(file, maxBytes = 65535)

Arguments

Value

Empty string to mimic the returned content of file.

Examples

notebookutils.fs.head("/tmp/my-folder/my-file")
notebookutils.fs.head("/tmp/my-folder/my-file", 1000)

notebookutils.fs provides utilities for working with various FileSystems.

Description

Below is overview about the available methods:

Usage

notebookutils.fs.help(methodName = "")

Arguments

Details

notebookutils.fs.cp: Copies a file or directory, possibly across FileSystems notebookutils.fs.mv: Moves a file or directory, possibly across FileSystems notebookutils.fs.ls: Array -> Lists the contents of a directory notebookutils.fs.mkdirs: Creates the given directory if it does not exist, also creating any necessary parent directories notebookutils.fs.put: Writes the given String out to a file, encoded in UTF-8 notebookutils.fs.head: Returns up to the first 'maxBytes' bytes of the given file as a String encoded in UTF-8 notebookutils.fs.append: Append the content to a file notebookutils.fs.rm: Removes a file or directory

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils fs module when used in azure synapse runtime.

Examples

notebookutils.fs.help()
notebookutils.fs.help("ls")

Lists the contents of a directory.

Description

Lists the contents of a directory.

Usage

notebookutils.fs.ls(dir)

Arguments

Value

Empty list to mimic the file list under dir.

Examples

notebookutils.fs.ls("/tmp/my-folder/")

Creates the given directory if it does not exist, also creating any necessary parent * directories.

Description

Creates the given directory if it does not exist, also creating any necessary parent * directories.

Usage

notebookutils.fs.mkdirs(dir)

Arguments

Value

FALSE to mimic the result if dir creation fail.

Examples

notebookutils.fs.mkdirs("/tmp/a/b/c")

Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)

Description

Attach remote storage (Blob, Gen2, Azure File Share) to all working nodes (driver node and worker nodes)

Usage

notebookutils.fs.mount(source, mountPoint, extraConfigs = NULL)

Arguments

Value

FALSE to mimic the result if mountPoint creation fail.

Examples

notebookutils.fs.mount("abfss://xxx.dfs.core.windows.net", "/mnt")

Attach remote storage (Blob, Gen2, Azure File Share) to driver node

Description

Attach remote storage (Blob, Gen2, Azure File Share) to driver node

Usage

notebookutils.fs.mountToDriverNode(source, mountPoint, extraConfigs = NULL)

Arguments

Value

TRUE if the path was successfully mounted.

Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.

Description

Show information about what is mounted. Any credentials used to mount the mount points listed will not be displayed.

Usage

notebookutils.fs.mounts(extraConfigs = NULL)

Arguments

Value

The list of MountPointInfo.

Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.

Description

Moves a file or directory, possibly across FileSystems. For intra-FileSystem, it is implemented by hadoop fs rename operation. For inter-FileSystem, This is implemented as a copy followed by delete.

Usage

notebookutils.fs.mv(from, to, createPath = FALSE, overwrite = FALSE)

Arguments

Value

FALSE to mimic the result of mv operation fail.

Examples

notebookutils.fs.mv("/tmp/my-folder/", "adls:/xxx/tmp/b")

Writes the given String out to a file, encoded in UTF-8.

Description

Writes the given String out to a file, encoded in UTF-8.

Usage

notebookutils.fs.put(file, content, overwrite = FALSE)

Arguments

Value

FALSE to mimic the result of file put operation fail.

Examples

notebookutils.fs.put("/tmp/my-file", "Hello world!", TRUE)

Refresh workspace level mount points.

Description

Refresh workspace level mount points.

Usage

notebookutils.fs.refreshMounts()

Value

FALSE to mimic the refreshMounts fail to refresh mount info.

Examples

notebookutils.fs.refreshMounts()

Removes a file or directory.

Description

Removes a file or directory.

Usage

notebookutils.fs.rm(dir, recurse = FALSE)

Arguments

Value

FALSE to mimic the result of dir deletion fail.

Examples

notebookutils.fs.rm("/tmp/my-folder/", TRUE)

Removes a mount point.

Description

Removes a mount point.

Usage

notebookutils.fs.unmount(mountPoint)

Arguments

Value

FALSE to mimic the result of unmount mountPoint fail.

Examples

notebookutils.fs.unmount("/mnt")

Removes a mount point from driver node.

Description

Removes a mount point from driver node.

Usage

notebookutils.fs.unmountFromDriverNode(mountPoint)

Arguments

Value

TRUE if the mount point was successfully unmounted.

Get help message for this module.

Description

Get help message for this module.

Usage

notebookutils.help(methodName = "")

Arguments

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils module when used in azure synapse runtime.

Examples

notebookutils.help()

Create a lakehouse

Description

Create a lakehouse

Usage

notebookutils.lakehouse.create(
  name,
  description = "",
  definition = "",
  workspaceId = ""
)

Arguments

Value

A lakehouse object

Delete a lakehouse

Description

Delete a lakehouse

Usage

notebookutils.lakehouse.delete(name, workspaceId = "")

Arguments

Get a lakehouse

Description

Get a lakehouse

Usage

notebookutils.lakehouse.get(name = "", workspaceId = "")

Arguments

Value

A lakehouse object

Get the definition of a lakehouse

Description

Get the definition of a lakehouse

Usage

notebookutils.lakehouse.getDefinition(name, workspaceId = "")

Arguments

Value

The definition of the lakehouse

Get the info of a Lakehouse with properties.

Description

Get the info of a Lakehouse with properties.

Usage

notebookutils.lakehouse.getWithProperties(name, workspaceId = "")

Arguments

Value

Artifact object. Please refer to: https://learn.microsoft.com/en-us/rest/api/fabric /articles/item-management/properties/lakehouse-properties

The lakehouse module.

Description

notebookutils.lakehouse.create(name: String, description: String, workspaceId: String): Lakehouse -> Create a lakehouse notebookutils.lakehouse.get(name: String, workspaceId: String): Lakehouse -> Get a lakehouse notebookutils.lakehouse.delete(name: String, workspaceId: String): void -> Delete a lakehouse notebookutils.lakehouse.update(name: String, newName: String, description: String, workspaceId: String): Lakehouse -> Update a lakehouse

Usage

notebookutils.lakehouse.help(methodName = "")

Arguments

List all lakehouses

Description

List all lakehouses

Usage

notebookutils.lakehouse.list(workspaceId = "", maxResults = 1000L)

Arguments

Value

A list of lakehouse objects

List all tables in a Lakehouse.

Description

List all tables in a Lakehouse.

Usage

notebookutils.lakehouse.listTables(
  lakehouse = "",
  workspaceId = "",
  maxResults = 1000L
)

Arguments

Value

A list of table objects

Starts a load table operation.

Description

Starts a load table operation.

Usage

notebookutils.lakehouse.loadTable(
  loadOption,
  table,
  lakehouse = "",
  workspaceId = ""
)

Arguments

Value

boolean

Update a lakehouse

Description

Update a lakehouse

Usage

notebookutils.lakehouse.update(
  name,
  newName,
  description = "",
  workspaceId = ""
)

Arguments

Value

A lakehouse object

Get the definition of a lakehouse

Description

Get the definition of a lakehouse

Usage

notebookutils.lakehouse.updateDefinition(name, definition, workspaceId = "")

Arguments

Value

The definition of the lakehouse

Create a notebook

Description

Create a notebook

Usage

notebookutils.notebook.create(
  name,
  description = "",
  content = "",
  defaultLakehouse = "",
  defaultLakehouseWorkspace = "",
  workspaceId = ""
)

Arguments

Value

A notebook object

Delete a notebook

Description

Delete a notebook

Usage

notebookutils.notebook.delete(name, workspaceId = "")

Arguments

This method lets you exit a notebook with a value.

Description

This method lets you exit a notebook with a value.

Usage

notebookutils.notebook.exit(value)

Arguments

Value

No return value, mimic behavior to set the notebook run exit value using value.

Examples

notebookutils.notebook.exit('exitVal')

Get a notebook

Description

Get a notebook

Usage

notebookutils.notebook.get(name, workspaceId = "")

Arguments

Value

A notebook object

The notebook module.

Description

The notebook module.

Usage

notebookutils.notebook.help(methodName = "")

Arguments

Value

No return value, print empty string to mimic the behavior of help method of mssparkutils notebook module when used in azure synapse runtime.

Examples

notebookutils.notebook.help()
notebookutils.notebook.help("run")

List all notebooks

Description

List all notebooks

Usage

notebookutils.notebook.list(workspaceId = "", maxResults = 1000L)

Arguments

Value

A list of notebook objects

Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.

Description

Runs a notebook and returns its exit value. The notebook will run in the current livy session context by default.

Usage

notebookutils.notebook.run(path, timeoutSeconds = 90, arguments = NULL)

Arguments

Value

Empty string to mimic the exitVal set by mssparkutils.notebook.exit.

Examples

notebookutils.notebook.run('NB1')
notebookutils.notebook.run('NB1', 200)
notebookutils.notebook.run('NB1', 200, list("input"=30))

Update a notebook

Description

Update a notebook

Usage

notebookutils.notebook.update(
  name,
  newName,
  description = "",
  workspaceId = ""
)

Arguments

Value

A notebook object

Get the definition of a notebook

Description

Get the definition of a notebook

Usage

notebookutils.notebook.updateDefinition(
  name,
  content,
  defaultLakehouse = "",
  defaultLakehouseWorkspace = "",
  workspaceId = ""
)

Arguments

Value

The definition of the notebook

provide a way to make people can update the endpoint

Description

provide a way to make people can update the endpoint

Usage

notebookutils.notebook.updateNBSEndpoint(endpoint)

Arguments

Get runtime properties

Description

Get runtime properties

Usage

notebookutils.runtime.context()

Value

A dummy env object to mimic the result of runtime context method when used in azure synapse runtime.

Examples

notebookutils.runtime.context()

notebookutils.runtime is a utility to manage runtime context. context() returns the runtime context as a list.

Description

notebookutils.runtime is a utility to manage runtime context. context() returns the runtime context as a list.

Usage

notebookutils.runtime.help(methodName = "")

Arguments

Set runtime high concurrency mode repl id

Description

Set runtime high concurrency mode repl id

Usage

notebookutils.runtime.setHcReplId(replId)

Arguments

Stop an interactive session

Description

Stop an interactive session

Usage

notebookutils.session.stop(detach = TRUE)

Arguments

Create a warehouse

Description

Create a warehouse

Usage

notebookutils.warehouse.create(
  name,
  description = "",
  definition = "",
  workspaceId = ""
)

Arguments

Value

A warehouse object

Delete a warehouse

Description

Delete a warehouse

Usage

notebookutils.warehouse.delete(name, workspaceId = "")

Arguments

Get a warehouse

Description

Get a warehouse

Usage

notebookutils.warehouse.get(name, workspaceId = "")

Arguments

Value

A warehouse object

Get the definition of a warehouse

Description

Get the definition of a warehouse

Usage

notebookutils.warehouse.getDefinition(name, workspaceId = "")

Arguments

Value

The definition of the warehouse

The warehouse module.

Description

notebookutils.warehouse.create(name: String, description: String, workspaceId: String): warehouse -> Create a warehouse notebookutils.warehouse.get(name: String, workspaceId: String): warehouse -> Get a warehouse notebookutils.warehouse.delete(name: String, workspaceId: String): void -> Delete a warehouse notebookutils.warehouse.update(name: String, newName: String, description: String, workspaceId: String): warehouse -> Update a warehouse

Usage

notebookutils.warehouse.help(methodName = "")

Arguments

List all warehouses

Description

List all warehouses

Usage

notebookutils.warehouse.list(workspaceId = "", maxResults = 1000L)

Arguments

Value

A list of warehouse objects

Update a warehouse

Description

Update a warehouse

Usage

notebookutils.warehouse.update(
  name,
  newName,
  description = "",
  workspaceId = ""
)

Arguments

Value

A warehouse object

Get the definition of a warehouse

Description

Get the definition of a warehouse

Usage

notebookutils.warehouse.updateDefinition(name, definition, workspaceId = "")

Arguments

Value

The definition of the warehouse

Assign a workspace to a capacity

Description

Assign a workspace to a capacity

Usage

notebookutils.workspace.assignToCapacity(capacityId, workspaceId = "")

Arguments

Value

Boolean indicating success

Create a workspace

Description

Create a workspace

Usage

notebookutils.workspace.create(name, description = "", capacityId = "")

Arguments

Value

A workspace object

Delete a workspace

Description

Delete a workspace

Usage

notebookutils.workspace.delete(workspaceId)

Arguments

Get a workspace

Description

Get a workspace

Usage

notebookutils.workspace.get(name = "")

Arguments

Value

A workspace object

The workspace module.

Description

notebookutils.workspace.assignToCapacity(capacityId: String, workspaceId: String): Boolean -> Assign a workspace to a capacity notebookutils.workspace.create(name: String, description: String, capacityId: String): workspace -> Create a workspace notebookutils.workspace.delete(workspaceId: String): void -> Delete a workspace notebookutils.workspace.get(name: String): workspace -> Get a workspace notebookutils.workspace.unassignFromCapacity(workspaceId: String): void -> Unassign a workspace from a capacity notebookutils.workspace.update(workspaceId: String, newName: String, description: String): workspace -> Update a workspace

Usage

notebookutils.workspace.help(methodName = "")

Arguments

List all workspaces

Description

List all workspaces

Usage

notebookutils.workspace.list(maxResults = 1000L)

Arguments

Value

A list of workspace objects

List the specified artifacts in the workspace

Description

List the specified artifacts in the workspace

Usage

notebookutils.workspace.listArtifacts(
  artifactType,
  workspaceId = "",
  maxResults = 1000L
)

Arguments

Value

A list of artifact objects

Unassign a workspace from a capacity

Description

Unassign a workspace from a capacity

Usage

notebookutils.workspace.unassignFromCapacity(workspaceId)

Arguments

Update a workspace

Description

Update a workspace

Usage

notebookutils.workspace.update(workspaceId, newName, description = "")

Arguments

Value

Updated workspace object