File I/O and OS Operations Intermediate

Introduction

Go's standard library provides everything you need for file operations without third-party dependencies. The os package handles file creation, opening, and metadata. The bufio package provides buffered I/O for efficiency. The filepath package handles path manipulation cross-platform. Together, they cover reading config files, writing logs, walking directory trees, and managing temporary files — all common in interviews and production code.

Syntax & Usage

Opening and Creating Files

// Open for reading only
f, err := os.Open("data.txt")
if err != nil {
    return fmt.Errorf("opening file: %w", err)
}
defer f.Close()

// Create (or truncate) for writing
f, err := os.Create("output.txt")
if err != nil {
    return fmt.Errorf("creating file: %w", err)
}
defer f.Close()

// Full control: flags + permissions
f, err := os.OpenFile("app.log",
    os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
if err != nil {
    return fmt.Errorf("opening log file: %w", err)
}
defer f.Close()

Function	Equivalent OpenFile flags	Use Case
os.Open(name)	O_RDONLY	Read existing file
os.Create(name)	O_RDWR\|O_CREATE\|O_TRUNC	Create or overwrite file
os.OpenFile(...)	Custom flags	Append, exclusive create, etc.

Common flags: O_RDONLY, O_WRONLY, O_RDWR, O_APPEND, O_CREATE, O_TRUNC, O_EXCL.

Reading Files

Read entire file into memory (simplest, for small files):

data, err := os.ReadFile("config.json")
if err != nil {
    return err
}
fmt.Println(string(data))

Read with io.ReadAll (from any io.Reader):

f, err := os.Open("data.txt")
if err != nil {
    return err
}
defer f.Close()

data, err := io.ReadAll(f)
if err != nil {
    return err
}

Read line by line (for large files):

f, err := os.Open("server.log")
if err != nil {
    return err
}
defer f.Close()

scanner := bufio.NewScanner(f)
for scanner.Scan() {
    line := scanner.Text() // current line without \n
    processLine(line)
}
if err := scanner.Err(); err != nil {
    return fmt.Errorf("scanning file: %w", err)
}

Buffered reading with custom buffer size:

scanner := bufio.NewScanner(f)
buf := make([]byte, 0, 1024*1024) // 1 MB capacity
scanner.Buffer(buf, 1024*1024)     // max token size 1 MB

Writing Files

Write entire file at once (simplest):

data := []byte("Hello, World!\n")
err := os.WriteFile("output.txt", data, 0644)
if err != nil {
    return err
}

Buffered writing (for many small writes):

f, err := os.Create("output.txt")
if err != nil {
    return err
}
defer f.Close()

w := bufio.NewWriter(f)
for _, line := range lines {
    fmt.Fprintln(w, line)
}
if err := w.Flush(); err != nil {
    return fmt.Errorf("flushing buffer: %w", err)
}

Write with fmt.Fprintf (formatted output):

f, err := os.Create("report.csv")
if err != nil {
    return err
}
defer f.Close()

fmt.Fprintf(f, "name,age,score\n")
for _, r := range records {
    fmt.Fprintf(f, "%s,%d,%.2f\n", r.Name, r.Age, r.Score)
}

File Permissions

Unix-style permission bits (ignored on Windows, but still required in function signatures):

Permission	Octal	Meaning
0644	rw-r--r--	Owner read/write, others read
0755	rwxr-xr-x	Owner all, others read/execute
0600	rw-------	Owner read/write only
0700	rwx------	Owner all, no others

os.WriteFile("secret.key", keyData, 0600) // restricted to owner
os.MkdirAll("logs/archive", 0755)         // directory with execute bit

File Info with os.Stat

info, err := os.Stat("data.txt")
if err != nil {
    if os.IsNotExist(err) {
        fmt.Println("file does not exist")
        return nil
    }
    return err
}

fmt.Println("Name:", info.Name())       // "data.txt"
fmt.Println("Size:", info.Size())       // bytes
fmt.Println("Mode:", info.Mode())       // file permissions
fmt.Println("ModTime:", info.ModTime()) // last modified
fmt.Println("IsDir:", info.IsDir())     // true if directory

Check existence pattern:

func fileExists(path string) bool {
    _, err := os.Stat(path)
    return !os.IsNotExist(err)
}

Directory Traversal with WalkDir

filepath.WalkDir (Go 1.16+) is more efficient than the older filepath.Walk:

err := filepath.WalkDir("./project", func(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err // propagate errors (e.g., permission denied)
    }

    if d.IsDir() && d.Name() == "vendor" {
        return filepath.SkipDir // skip entire directory
    }

    if filepath.Ext(path) == ".go" {
        fmt.Println(path)
    }

    return nil
})
if err != nil {
    return fmt.Errorf("walking directory: %w", err)
}

Reading directory entries without recursion:

entries, err := os.ReadDir(".")
if err != nil {
    return err
}
for _, entry := range entries {
    fmt.Printf("%s (dir=%t)\n", entry.Name(), entry.IsDir())
}

Temporary Files and Directories

// Temporary file (in default temp dir, with pattern prefix)
tmpFile, err := os.CreateTemp("", "myapp-*.txt")
if err != nil {
    return err
}
defer os.Remove(tmpFile.Name()) // clean up when done
defer tmpFile.Close()

fmt.Fprintln(tmpFile, "temporary data")
fmt.Println("temp file:", tmpFile.Name()) // e.g., /tmp/myapp-123456.txt

// Temporary directory
tmpDir, err := os.MkdirTemp("", "myapp-")
if err != nil {
    return err
}
defer os.RemoveAll(tmpDir)

The * in the pattern is replaced with a random string. Pass "" as the first argument to use the system's default temp directory.

Working with Paths (filepath Package)

// Build paths cross-platform
path := filepath.Join("data", "users", "config.json")
// Linux: "data/users/config.json"
// Windows: "data\\users\\config.json"

// Split path components
dir := filepath.Dir("/home/user/doc.txt")   // "/home/user"
base := filepath.Base("/home/user/doc.txt")  // "doc.txt"
ext := filepath.Ext("photo.jpg")             // ".jpg"

// Resolve to absolute path
abs, err := filepath.Abs("./relative/path")

// Match patterns
matched, err := filepath.Match("*.go", "main.go") // true

// Clean up messy paths
clean := filepath.Clean("./a/../b/./c") // "b/c"

Environment Variables

// Get (returns "" if not set)
home := os.Getenv("HOME")

// Get with existence check
val, exists := os.LookupEnv("DATABASE_URL")
if !exists {
    log.Fatal("DATABASE_URL is required")
}

// Set (for current process only)
os.Setenv("APP_MODE", "production")

// Unset
os.Unsetenv("TEMP_TOKEN")

// All environment variables
for _, env := range os.Environ() {
    fmt.Println(env) // "KEY=VALUE" format
}

Quick Reference

Operation	Function	Notes
Read entire file	os.ReadFile(name)	Returns []byte
Write entire file	os.WriteFile(name, data, perm)	Creates or truncates
Open for reading	os.Open(name)	Returns *os.File
Create for writing	os.Create(name)	Truncates if exists
Open with flags	os.OpenFile(name, flag, perm)	Full control
Read line by line	bufio.NewScanner(r)	.Scan(), .Text()
Buffered write	bufio.NewWriter(w)	Don't forget .Flush()
File info	os.Stat(name)	Size, permissions, mod time
Walk directory	filepath.WalkDir(root, fn)	Recursive traversal
Read directory	os.ReadDir(name)	Non-recursive listing
Temp file	os.CreateTemp(dir, pattern)	Random name with pattern
Join paths	filepath.Join(parts...)	Cross-platform separator
Get env var	os.Getenv(key)	Returns "" if unset
Check env var	os.LookupEnv(key)	Returns (value, exists)

Best Practices

1. Always defer f.Close() immediately after opening a file — prevents resource leaks on any return path.
2. Use os.ReadFile/os.WriteFile for small files — simpler and less error-prone than manual open/read/close.
3. Use bufio.Scanner for large files — don't load gigabyte files into memory with ReadFile.
4. Use filepath.Join for paths — never concatenate with "/" or "\\" manually. Your code needs to work cross-platform.
5. Use filepath.WalkDir over filepath.Walk — WalkDir avoids a Stat call per entry and is faster.
6. Clean up temporary files — always defer os.Remove(tmpFile.Name()) or defer os.RemoveAll(tmpDir).
7. Use os.LookupEnv over os.Getenv to distinguish between "empty" and "not set".

Common Pitfalls

Not closing files in loops

// BUG: defer doesn't run until function returns — all files open at once
for _, path := range paths {
    f, err := os.Open(path)
    if err != nil { return err }
    defer f.Close() // piles up!
    process(f)
}

// FIX: close explicitly or extract to helper
for _, path := range paths {
    f, err := os.Open(path)
    if err != nil { return err }
    process(f)
    f.Close()
}

Forgetting to flush bufio.Writer

w := bufio.NewWriter(f)
w.WriteString("important data")
// BUG: data stuck in buffer — never written to file!

// FIX: always flush
if err := w.Flush(); err != nil {
    return err
}

Closing the underlying file does not flush the bufio.Writer buffer.

Scanner default buffer size

// BUG: lines longer than 64 KB cause scanner.Err() to return an error
scanner := bufio.NewScanner(f)
for scanner.Scan() { ... }

// FIX: increase buffer size if needed
scanner.Buffer(make([]byte, 0, 1024*1024), 1024*1024)

The default max token size is 64 KB (bufio.MaxScanTokenSize).

Ignoring Close() error on writes

f, _ := os.Create("data.txt")
f.Write(data)
f.Close() // Close flushes to disk — this can fail!

// FIX: check the error
if err := f.Close(); err != nil {
    return fmt.Errorf("closing file: %w", err)
}

For read-only files, ignoring Close() error is acceptable. For writes, Close() can fail if the final flush to disk fails.

Using os.IsNotExist instead of errors.Is

// Works but is the older pattern
if os.IsNotExist(err) { ... }

// Preferred since Go 1.13 — works with wrapped errors
if errors.Is(err, os.ErrNotExist) { ... }

Performance Considerations

• os.ReadFile vs bufio.Scanner: ReadFile loads everything into memory — fast for small files, dangerous for large ones. Use bufio.Scanner or chunked reads for files larger than available memory.
• bufio.Writer reduces syscalls: Each Write() call to os.File is a syscall. bufio.Writer batches small writes into a single 4 KB (default) buffer, reducing syscall overhead by orders of magnitude.
• filepath.WalkDir vs filepath.Walk: WalkDir avoids calling os.Stat on every entry (the DirEntry provides Type() without a stat call). For directories with thousands of entries, this is measurably faster.
• os.ReadDir vs os.ReadDir + sort: os.ReadDir returns entries sorted by name. If you don't need sorted output, os.Open + f.ReadDir(-1) can be slightly faster.
• Temporary files: os.CreateTemp creates files in the OS temp directory which may be on a faster filesystem (tmpfs on Linux). Use this for scratch data that doesn't need persistence.

Interview Tips

Interview Tip

"How do you read a large file efficiently in Go?" Use bufio.Scanner to read line by line, or io.Reader with a fixed-size buffer to read in chunks. Never use os.ReadFile or io.ReadAll for large files — they load everything into memory. For structured data, use a streaming decoder (e.g., json.NewDecoder).

Interview Tip

"What's the idiomatic way to handle file cleanup?" Open the file, check the error, then immediately defer f.Close(). This ensures the file is closed regardless of how the function exits — normal return, early error return, or panic. It's the standard three-line pattern in Go.

Interview Tip

"How do you safely write to a file?" For atomic writes, write to a temporary file in the same directory, then rename it to the target path. os.Rename is atomic on most filesystems. This prevents partial writes from corrupting the target file if the process crashes mid-write.

Interview Tip

"What's the difference between os.Getenv and os.LookupEnv?" Getenv returns an empty string for both "not set" and "set to empty". LookupEnv returns a second boolean indicating whether the variable exists, letting you distinguish the two cases.

Key Takeaways

• Use os.ReadFile/os.WriteFile for simple, small-file operations.
• Use bufio.Scanner for line-by-line reading of large files; use bufio.Writer for efficient writes.
• Always defer f.Close() immediately after opening — and check the error on write paths.
• Use filepath.Join for cross-platform path construction — never hardcode separators.
• filepath.WalkDir is the modern, efficient choice for recursive directory traversal.
• Clean up temporary files with defer os.Remove() or defer os.RemoveAll().
• Use os.LookupEnv when you need to distinguish "not set" from "empty".