U-SQL adds support for computed file property columns on EXTRACT statement. Sometimes customers would like to get information about the files that they process, such as the full URI path or information about size, creation or modification dates. Likewise customers would Read More …
Category: Azure
U-SQL Table
Azure Data Lake Analytics (U-SQL) originates from the world of Big Data, in which data is processed in a scale-out manner by using multiple nodes. These nodes can access the data in several formats, from flat files to U-SQL tables. Read More …
Data Factory Ingestion Framework: Part 1 – Schema Loader
Use Case A business wants to utilize cloud technology to enable data science and augment data warehousing by staging and prepping data in a data lake. There are multiple different systems we want to pull from, both in terms of Read More …
T-SQL TO U-SQL DATA TYPE CONVERSION
When working with code generated solutions we often need to convert datasets from SQL Server (T-SQL) data types to Azure Data Lake Analytics (U-SQL) data types. As you probably know U-SQL has a hybrid syntax of T-SQL and C# which Read More …
Part 9: Extending U-SQL
There are 5 kinds of User-Defined entities in U-SQL User-Defined Functions (UDFs) User-Defined Types (UDTs) User-Defined Aggregators (UDAggs) User-defined Operators (UDOs) User-Defined Appliers All of them are defined by .NET code. C# is not required. Any .NET language will work. User-Defined Functions User defined functions are normal static methods Read More …
Part 8: Set operations and Joins
Set operations are a way of merging rowsets together based on set theoretic operations such as union (UNION). intersection (INTERSECT), complement (EXCEPT). Sample data Let’s define two RowSets: @a and @b. Notice that both RowSets have duplicate rows. UNION UNION combines two rowsets. UNION Read More …
Part 7: Window Functions
Window functions were introduced to the ISO/ANSI SQL Standard in 2003. U-SQL adopts a subset of window functions as defined by the ANSI SQL Standard. Window functions are used to do computation within sets of rows called windows. Windows are defined Read More …
Part 6: The U-SQL Catalog and Assemblies
The U-SQL Catalog is the way U-SQL organizes data and code for re-use and sharing. Catalog organization Every ADLA account has a single U-SQL catalog. The catalog cannot be deleted. Each U-SQL catalog contains one or more U-SQL databases. Every Read More …
Part 5: Working with FileSets
Reading and Writing Files: Built-in Extractors U-SQL has three built-in extractors that handle text Extractors.Csv() reads comma-separated value (CSV) Extractors.Tsv() reads tab-separated value (TSV) Extractors.Text() reads delimited text files. Extractors.Csv and Extractors.Tsv are the same as Extractors.Text but they default to a specific delimiter appropriate for the format Read More …
Part 4: U-SQL Expressions
Overview Clauses such as SELECT, WHERE, and HAVING (among others) allow you to enter U-SQL expressions. An expression in a programming language is a combination of explicit values, constants, variables, operators, and functions that are interpreted according to the particular rules of precedence and Read More …