The Curse of Dimensionality

Explains the fundamental differences between tensor dimensions, size, and shape, clarifying frequent misconceptions—such as the distinction between the number of features (“columns”) and true data dimensions—while also demystifying reshaping operations like expand_dims, squeeze, and transpose in NumPy. Through practical examples from images and natural language processing, listeners learn how to manipulate tensors to match model requirements, including scenarios like adding dummy dimensions for grayscale images or reordering axes for sequence data.

Links

• Notes and resources at ocdevel.com/mlg/mla-5
• Try a walking desk stay healthy & sharp while you learn & code

Definitions

• Tensor: A general term for an array of any number of dimensions.

  • 0D Tensor (Scalar): A single number (e.g., 5).
  • 1D Tensor (Vector): A simple list of numbers.
  • 2D Tensor (Matrix): A grid of numbers (rows and columns).
  • 3D+ Tensors: Higher-dimensional arrays, such as images or batches of images.

• NDArray (NumPy): Stands for "N-dimensional array," the foundational array type in NumPy, synonymous with "tensor."

Tensor Properties Dimensions

• Number of nested levels in the array (e.g., a matrix has two dimensions: rows and columns).
• Access in NumPy: Via .ndim property (e.g., array.ndim).

Size

• Total number of elements in the tensor.
• Examples:
  • Scalar: size = 1
  • Vector: size equals number of elements (e.g., 5 for [1, 2, 3, 4, 5])
  • Matrix: size = rows × columns (e.g., 10×10 = 100)
• Access in NumPy: Via .size property.

Shape

• Tuple listing the number of elements per dimension.
• Example: An image with 256×256 pixels and 3 color channels has shape = (256, 256, 3).

Common Scenarios & Examples Data Structures in Practice

• CSV/Spreadsheet Example: Dataset with 1 million housing examples and 50 features:
  • Shape: (1_000_000, 50)
  • Size: 50,000,000
• Image Example (RGB): 256×256 pixel image:
  • Shape: (256, 256, 3)
  • Dimensions: 3 (width, height, channels)
• Batching for Models:
  • For a convolutional neural network, shape might become (batch_size, width, height, channels), e.g., (32, 256, 256, 3).

Conceptual Clarifications

• The term "dimensions" in data science often refers to features (columns), but technically in tensors it means the number of structural axes.
• The "curse of dimensionality" often uses "dimensions" to refer to features, not tensor axes.

Reshaping and Manipulation in NumPy Reshaping Tensors

• Adding Dimensions:

  • Useful when a model expects higher-dimensional input than currently available (e.g., converting grayscale image from shape (256, 256) to (256, 256, 1)).
  • Use np.expand_dims or array.reshape.

• Removing Singleton Dimensions:

  • Occurs when, for example, model output is (N, 1) and single dimension should be removed to yield (N,).
  • Use np.squeeze or array.reshape.

• Wildcard with -1:

  • In reshaping, -1 is a placeholder for NumPy to infer the correct size, useful when batch size or another dimension is variable.

• Flattening:

  • Use np.ravel to turn a multi-dimensional tensor into a contiguous 1D array.

Axis Reordering

• Transposing Axes:
  • Needed when model input or output expects axes in a different order (e.g., sequence length and embedding dimensions in NLP).
  • Use np.transpose for general axis permutations.
  • Use np.swapaxes to swap two specific axes but prefer transpose for clarity and flexibility.

Practical Example

• In NLP sequence models:
  • 3D tensor with (batch_size, sequence_length, embedding_dim) might need to be reordered to (batch_size, embedding_dim, sequence_length) for certain models.
  • Achieved using: array.transpose(0, 2, 1)

Core NumPy Functions for Manipulation

• reshape: General function for changing the shape of a tensor, including adding or removing dimensions.
• expand_dims: Adds a new axis with size 1.
• squeeze: Removes axes with size 1.
• ravel: Flattens to 1D.
• transpose: Changes the order of axes.
• swapaxes: Swaps specified axes (less general than transpose).

Summary Table of Operations Operation NumPy Function Purpose Add dimension np.expand_dims Convert (256,256) to (256,256,1) Remove dimension np.squeeze Convert (N,1) to (N,) General reshape np.reshape Any change matching total size Flatten np.ravel Convert (a,b) to (a*b,) Swap axes np.swapaxes Exchange positions of two axes Permute axes np.transpose Reorder any sequence of axes Closing Notes

• A deep understanding of tensor structure - dimensions, size, and shape - is vital for preparing data for machine learning models.
• Reshaping, expanding, squeezing, and transposing tensors are everyday tasks in model development, especially for adapting standard datasets and models to each other.