Linux Desktop Control

Automate and control your Linux desktop using command-line tools. Capture screenshots, control mouse/keyboard, and manage windows.

When to Use

• Take screenshots of screen or specific windows
• Automate mouse movements and clicks
• Simulate keyboard input
• Manage and interact with windows
• Build desktop automation workflows

Prerequisites

Install required tools:

sudo apt-get install scrot xdotool xclip x11-utils x11-apps

Quick Start

Take Screenshot

python3 scripts/linux-desktop.py screenshot
# Output: ~/screenshot_20240224_203901.png

List Windows

python3 scripts/linux-desktop.py list

Move Mouse and Click

python3 scripts/linux-desktop.py move 500 300
python3 scripts/linux-desktop.py click

Type Text

python3 scripts/linux-desktop.py type "Hello World"

Commands

screenshot [path]

Capture a screenshot of the entire screen.

Examples:

# Save to default location (/tmp/screenshot_YYYYMMDD_HHMMSS.png)
python3 scripts/linux-desktop.py screenshot

# Save to custom path
python3 scripts/linux-desktop.py screenshot ~/desktop.png

window [window_id] [path]

Capture a screenshot of a specific window.

Examples:

# Screenshot active window
python3 scripts/linux-desktop.py window

# Screenshot specific window
python3 scripts/linux-desktop.py window 0x12345678 ~/window.png

active

Get information about the currently active window.

python3 scripts/linux-desktop.py active
# Output: 🖥️ Active Window
#         ID: 0x12345678
#         Title: Terminal

list

List all visible windows.

python3 scripts/linux-desktop.py list
# Output: 🪟 Found 5 windows:
#         1. 0x12345678 - Terminal
#         2. 0x87654321 - Chrome

move <x> <y>

Move mouse cursor to specified coordinates.

Examples:

python3 scripts/linux-desktop.py move 100 200
# Moves mouse to (100, 200)

python3 scripts/linux-desktop.py move 500 300
# Moves mouse to center of 1000x600 area

click [button]

Click mouse button at current cursor position.

Button values:

• 1 - Left button (default)
• 2 - Middle button
• 3 - Right button

Examples:

python3 scripts/linux-desktop.py click
# Left click

python3 scripts/linux-desktop.py click 3
# Right click

type <text>

Type text at current cursor position (must be in focused window).

Examples:

python3 scripts/linux-desktop.py type "Hello World"

python3 scripts/linux-desktop.py type "ls -la"

python3 scripts/linux-desktop.py type "sudo apt update"

key <keyspec>

Press keyboard keys.

Common keys:

• Return - Enter key
• Escape - Escape key
• Tab - Tab key
• BackSpace - Backspace
• Delete - Delete
• Up, Down, Left, Right - Arrow keys
• Home, End, Page_Up, Page_Down
• F1 through F12
• Ctrl+c, Ctrl+v, Ctrl+a, Ctrl+z - Key combinations

Examples:

python3 scripts/linux-desktop.py key Return

python3 scripts/linux-desktop.py key Escape

python3 scripts/linux-desktop.py key Ctrl+a

python3 scripts/linux-desktop.py key F5

screen

Get screen information.

python3 scripts/linux-desktop.py screen
# Output: 🖥️ Screen Info
#         Resolution: 1920x1080

Automation Examples

Basic Automation

# Move mouse, click, type, and press enter
python3 scripts/linux-desktop.py move 100 100
python3 scripts/linux-desktop.py click
python3 scripts/linux-desktop.py type "ls -la"
python3 scripts/linux-desktop.py key Return

Web Search Automation

# Open browser, navigate to Google, search
python3 scripts/linux-desktop.py move 100 50
python3 scripts/linux-desktop.py click
python3 scripts/linux-desktop.py type "https://www.google.com"
python3 scripts/linux-desktop.py key Return
sleep 2
python3 scripts/linux-desktop.py type "how to make money online"
python3 scripts/linux-desktop.py key Return

Screenshot Workflow

# Take screenshot before and after action
python3 scripts/linux-desktop.py screenshot /tmp/before.png
python3 scripts/linux-desktop.py key F5  # Refresh
sleep 1
python3 scripts/linux-desktop.py screenshot /tmp/after.png

Tips

• Always check the active window before typing
• Use sleep commands between actions for reliability
• Take screenshots to verify state changes
• Test commands one by one before building complex workflows
• Use window list to find specific window IDs for targeting

Troubleshooting

"Command not found" errors:

sudo apt-get install scrot xdotool xclip x11-utils x11-apps

Permission denied:

• Ensure you're running in a graphical session (X11 or Wayland)
• Some actions require focus on the target window

Mouse doesn't move:

• Check if another application is grabbing the mouse
• Try moving the mouse manually to see if it's responsive

Security Notes

• This skill can control your desktop - use with caution
• Don't automate sensitive actions without verification
• Always review automation scripts before running